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ABSTRACT 

The Reflective Appraisal of Programs (RAP) approach 
allows county extension staff (in cooperation with volunteer leaders, 
specialists, and district staff) to obtain systematic evidence on 
results that participants perceive to have occurred in the months or 
years following their involvement in an extension program. Simpler 
than a "cookbook" for documenting results of extension programs, RAP 
resembles a "package mix." RAP contains standard components that can 
be easily adjusted or added to in order to create a study tailored to 
specific needs. Standardised interview guest ibns used in a RAP study 
are applicable to the clientele of almost any extension program. This 
"RAP package" contains an introduction to the program and three 
publications. The first publication provides a rationale, featuring 
RAP's unique features and its relation to other approaches for 
evaluating extension programs. The second and third publications 
include a guide and an accompanying workbook, which present 
step-by-step instructions and planning aids for implementing a RAP 
study. (AH) 
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Overview of Reflective Appraisal 
of Programs (RAP)' 

People who make or influence decisions regarding the 
direction and resources of extension programs in coun- 
ties have a growing n$ed for systematic evidence on the 
results of the programs. Using the Reflective Appraisal of 
Programs (RAP) approach, county extension staff (in 
cooperation with volunteer leaders, specialists, and 
district staff) obtain such systematic evidence by docu- 
menting the results that participants perceive to have 
occurred in the months and years following their involve- 
ment in an extension program. 

The RAP approach has already been used to study a 
wide range of extension programs in counties, including: 

• Swine disease control program 

• Community recreation education program 

• 4-H environmental education program 

• Summer telephone advisory program on fruit and 

vegetable preservation 

• Homemaker club program 

• Integrated pest management program 

• Consumer education program 

• Teenage sex-education program 

• 4-H home fire-prevention program » 

• Handicraft- marketing program 

Simpler than a "cookbook" for documenting the results; 
of extension programs, RAP resembles a 'package mix " 
That is, RAP contains standard components that can be 
easily adjusted or added to in order to create a study 
tailored to specific needs. The standardized interview 
questions used in a RAP study are applicable to the clien- 
tele of almost any^extension program. By "plugging" 
selected program activities into these questions, exten- 
sion field staff obtain participants' perceptions of the 
results of a program in which they were involved, 



How Does RAP Work? 

• RAP relies on perceptions or "reflective" evidence on 
the results of the program being studied. Program partici- 
pants estimate (reflect upon) the extent to which a 
program brought about change and "payoff ," 

• Interviews are conducted, usually by telephone, with 
a minimum of 30-40 program participants per county. 
RAP can be used to study the results of extension 
programs in extension areas and districts as well as in 
counties. 

• RAP uses standardized interview questions that can 
be adapted to extension programs on a wide variety of 
subject matters and using a variety of educational 
methods, 

• In leading a RAP study, a county extension staff 
member needs to expend a total of 40-50 hours. 

Who Implements a RAP Study? 

We suggest that county extension staff involve volun- 
teer extension leaders and extension district, regional, 
area, and state staff in the planning and implementation 
of their RAP studies. Included in this RAR team should be 
people who make or influence decisions about the 
program being studied. 

.> 

What Are the Steps of a RAP Study? 

As a RAP team conducts a study, they follow the six 
stSps listed below, The RAP guidebook and workbook 
help "guide" them through this process. 

1 Select a program for study, 

2. Prepare a description of both the activities of the 
program selected and a complete list of who partici- 
pated in the program during the past months to 

years* 

3. Select specific levels of evidence (as defined below) 
to be collected regarding the results of the program. 

4. Interview program participants or a sample of the 
participants on what they perceive to be the results 
and value of the program. 

5. Use the findings of the interviews to draw conclusions 
about and appraise (evaluate) results of the program. 

6. Recommend how decision makers can use the find- 
ings, conclusions, and appraisals, 
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Sample Interview Questions 

A program participant being interviewed is first 
reminded of the activities (educational methods and 
topics) of the program being studied. Interviewees then 
indicate the extent to which they participated in these 
activities and respond to standardized questions on the 
results of the program. RAP can help get evidence on 
program results at thetoilowing levels: reactions to 
program activities; KASA change (knowledge, attitude, 
•kill, and aspiration change); practice change; and end 
results of KASA change or practice change-. r 

The following is a list of possible educational methods 
and subjects from programs on energy conservation. By 
"plugging" these examples into the sample RAP interview 
items in this publication, you will be able to better under- 
stand how the interview questions can be adapted to 
virtually any extension program, 

Horn* economics 

Methods: meetings, newsletters, TV spots 
Subjects: purchase of energy-efficient household 
appliances 

Agriculture 

Methods: demonstrations, interactive computer with 
videotext, farm visits 

Subjects: solar grain-drying methods, machinery 
Community development 

Methods: regional meetings on analyses of public records 
Subjects: intercommunity cooperation in cutting costs of 
ambulance services 

4-H Youth 

Methods: club meetings, individual projects 

Subjects: daytime and nighttime temperature control for 

home heating 

The following question is designed to measure partici- 
pants' reactions to a program's activities. 

i 

Say to the interviewee: 

To what extent did the (educational method) on 
(program topic) meet your expectations at the time? 

Then read the interviewee the following answers and 
have him or her choose the answer that most nearly 
describes the way he or she feels. 

to a great extent 

, to a fair extent 

to a slight extent 

nof at all 

Check whichever response the interviewee provides. 

Provide a space to indicate if the person answers "don't 
know" or "don't recall." 
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Try to find out specifically what the interviewee meant 
by his or her response by following up with a probe 
(open-ended) question such as either of the following: 

• Please explain your answer a little more fully. 

• Would you give me an example of what you mean by 
your answer. * m * - 

The following question is designed to measure partici- 
pants' practice change (application of the content of the 
program in which they participated). 

Say to the interviewee: 

To what extent have you put to use the ideas or skills 
you learned regarding (program topic)? 

Then read the interviewee the following answers and 
have him or her choose the answer that most nearly 
describes the way he or she feels, 

_ to a great extent 

to a fair extent 

to a slight extent 

■— not at all 

Check whichever response the interviewee provides. 

Again, find out what the interviewee meant by his or 
her response by following up with a probe question such 
as^either of the following: 

• Please explain what you mean a little more fully. 

• Would you give me an example of what you mean. 

The above format can also be used for interview ques- 
tions at the other levels of evidence to find out how much 
clientele have»learned through their participation in the 
program, the positive or negative effects of applying what 
they learned, etc, 

The wording of the questions can be modified as 
necessary, and other types of questions can, of course, 
be included (for example, how participants became aware 
of the program and their reasons for participating). 



The "RAP Package" 

The "RAP package" contains the following three publi- 
cations: 

Reflective Appraisal of Programs (RAP): An Approach 
to Studying Clientele-Perceived Results of Cooperative 
Extension Programs— Rationale, (Presents RAP's unique 
features and its relation to other approaches for 
evaluating extension programs.) 

Reflective Appraisal of Programs (RAP): An Approach 
to Studying Clientele-Perceived Results of Cooperative 
Extension Programs-^Qutde and Workbook. (These 
present step-by-step instructions and planning aids for 
implementing a RAP study.) 

RAP Is based in part on two USDA publications by 
Claude F. Bennett: Analyzing Impacts of Extension 
Programs, ESC-575 (1976) and Teaching Materials on 
"Seven Levels of EWdence": A Guide for Extension \ 
Workers, ESC-575 (1980). 1 



Additional copies of the RAP package may be ordered 
for $2.00 (New York State residents) or $2.50 (out-of- 
state residents) plus postage and handling. Minimum 
order $10.00. Address all inquiries to: 

Media Services f 
B-10 Renssalaer Hall 
Cornell University 
Ithaca, NY 14853 
Tel: (607) 256-3157 
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This publication isTntended for extension administra- 
tors, program staff, and evaluation staff who wish to 
understand why extension staff are being encouraged to 
use the Reflective Appraisal of Programs (RAP) approach 
to study the results of extension programs. The rationale 
provided herein may be especially useful to state, district, 
and county directors of extension who set policies for 
extension program evaluation efforts. Companion publica- 
tions'in the "RAP package"— a guide and accompanying 
workbook— present step-by-step instructions Bnd "plan- 
ning aids" for extension staff who wish to implement a, 
RAP study. RAP is based on the premise that county 
extension staff shpuld take a leading role in studies of 
extension program results irr counties 

This publication focuses primarily on the strengths of 
one approach— Reflective Appraisal of Programs (RAP)— 
for determining and appraising (evaluating) results of 
extension programs in counties, A secondary focus is on 
the validity of perceptual or reflective data. Comments on 
how extension staff can be trained in the RAP approach 
also are included, 

RAP is only one of several possible approaches to 4 
formal program evaluation within state extension services. 
Other strategies include, for example, state-level studies 
of extension program results (as developed recently in 
Wisconsin and Ohio); a combination of state and county 
studies (as developed in Michigan, North Carolina* and 
West Virginia): and county extenson program reviews (as 
developed in Florida), 

State program stdff may find this publication helpful in 
determining how they might assist county extension staff 
in evaluating program results. State program evaluation 
specialists may find the publication useful in clarifying 
and modifying their role in helping state and county 
program staff with extension program evaluation, 



RAP was inspired by Patrick Borich of the University of 
Minnesota, who has persistently challenged evaluation 
specialists in Cooperative Extension to enable county 
* extension staff to evaluate their programs. 

Development of RAP was encouraged and aided by 
students and participants in extension-staff development 
classes and workshops held at the following locations: 
University of Missouri, Columbia, Missouri (1977); 
National 4-H Center, Chevy Chase, Maryland (1978); 
University of Minnesota, Duluth, Minnesota (1979); Ohio 
State University, Columbus, Ohio (1979); North Carolina 
State University, Raleigh, North Carolina (1980); National 
Association of 4-H Extensiqn Agents Conference at 
Detroit, Michigan (1980); and University of Arizona, 
Tucson, Arizona (1981), 

David Deshler, Peter Warnock, and Carolyn Boegly of 
Cornell University envisioned a potential use for RAP by 
Cooperative Extension in New York State in December 
1980, members of Cornell program teams (extension 
representatives and program coordinators) and Office of 
Director staff received training in RAP so that they could 
conduct RAP studies Jointly with selected county staff. 
Carol L Anderson, associate director of Cooperative 
Extension at Cornell University* coordinated efforts on 
these studies as a trial of RAP's suitability for statewide 
use by county extension staff in New York, 

Much credit and appreciation is due to reviewers of 
draft copies of RAP; they raised the quality of this publi- 
cation greatly through their incisive critiques and expert 
suggestions. Technical reviewers were Mary Andrews, 
Cooperative Extension Service, Michigan State University; 
Sue Cunningham, Cooperative Extension, Cornell Univer- 
sity; David Deshler, Cooperative Extension, Cornell 
University; Laverne Forest, University of Wisconsin- 
Extension: Constance McKenna, Extension Service, US. 
Department of Agriculture; Michael Patton, Minnesota 
Center "for Social Research, University of Minnesota; 
Kenneth Pigg, Department of Sociology, University, of, 
Kentucky;' Joan Wright, Agricultural Extension Service, 
North Carolina State University; and Bettie Lee Yerka, 
Cooperative Extension, Cornell University. * * 

Two books were especially stimulating and instructive 
In preparing RAP; Evaluation in Extension, edited by 
Darcie Byrn, and written by members of the Division of 
Extension Research and Training, Federal Extension 
Service, USDA (1965); and Utilization-Focused Evalua- 
tion, by Michael Ouinn Patton, Sage Publications, Inc. 
(1978), 

I appreciate the assistance of Erica Fox, of Media 
Services at Cornell University, for greatly improving the 
" r . clarity and readability of the RAP package. 

The author is grateful to Mrs. Gloria Robinson, who 
exhibited truly awesome perseverance and patience la 
typing the repeatedly revised drafts of this publication. 



Need for a Method to Study 
Results of County Extension 
Programs 

Public and private funding for extension programs can 
be justified in several ways. First, extension, like other 
organizations, makes promises to people who finance its 
programs. These promises generally are statements of 
need and associated goals that are included in extension, 
program and plan-of-work documents, budget justifica- 
tions to policy makers and legislators, and public rela- 
tions releases. 

Another way to justify funding is by claiming accom- 
plishments for past extension programs. Such claims 
generally are based On the casual observations of 
program personnej or on testimony from a few hand- 
picked program participants. Sometimes, such claims are 
based on evidence of improved social or economic condi- 
tions, which the program is assumed to have produced 
or helped produce. 

Promises and claims will continue to be important, 
especially in areas where fufiders already view extension 
favorably: legislators and polity makers who have had 
positive experiences regarding extension want to believe 
it is giving the public its money's worth. On the other 
hand, In areas where legislators and policy makers are 
unfamiliar with extension, or for some reason question its 
effectiveness, a third way to justify budgets is rapidly 
growing in importance: documented studies of the results 
of extension programs. Such studies are being used 
increasingly to meet funders' accountability requirements 
and to help extension develop improved programs. These 
scientific Studies generally are conducted by social scien- 
tists, program analysts, and evaluation specialists at state, 
multi-state, and national levels. At times, program staff 
conducj or participate in these studies. 

Studies of the results of nationwide extension programs 
are conducted for Congress and federal executives, while 
studiee of the results of statewide extension programs are 
conducted primarily for state legislators, state agencies, 
university administrators, and other interested parjies 
State and national studies usually are supported by 
special budgets and take months to conduct. 



Documenting Program Results 
in Counties 

How do studies of the results of extension programs in 
individual counties fit into the overall pattern of extension 
accountability and program improvement? Systematic 
evidence on the results of programs is apparently often 
needed at the county level. County agents need such 
evidence to modify programs and to be accountable 
within extension, but there is also growing pressure from 
county legislators and executives for credible, generaliz- 
able, clientele-based evidence on program results. In 
addition, some volunteer extension program-development 
committees are requesting accountability information. 
Pressure for county-level studies tends to increase as the 
proportion of extension funding supplied by county 
revenues increases, as the county becomes more urban- 
ized, and as public or private funding for special 
programs In the county increases, 

A recent national survey on program evaluation in 
extension obtained responses from a representative 
sample of 1.520 county extension agents. One finding 
showed that 29 percer^j of the county agents viewed 
formal evaluations as* ''useful for the purposes of accoun- 
tability reporting outside the extension organization M The m 
authors of the study commented that "in states in which 
county extension programs are heavily dependent upon 
funding from county sources there is a tendency for 
county staff members to be strongly aware of a need to 
be actively engagpd in formal program evaluation for the 
purposes of accountability. (These) county staff not 
only accept the need for formal, accountability-focused 
evaluation but are interested in being able to more effec- 
tively conduct such evaluation 

A related set of findings from the same survey indicated 
agents' average ratings of the usefulness of formal 
program equation for various purposes, On a five-point 
scale (5=very great extent. 4-great extent, 3-some extent. 
2-small extent. 1-rto extent), their average ratings were 
as follows: 

• for purpose of revising and improving existing or 
continuing programs (4 3) — * 

• for assessing new progrdfns (4,2) 

• for accountability reporting inside the extension organi- 
zation (3 8) j 

• \for accountability reporting outside the extension 
©ionization (3.6) 

• for assistance in administrative decision making (3 5) 

• for satisfying requirements of specially funded 
programs (3 3) 



James L Summers Robert W Miller Cecil E Carter. Richard E j 
Young, Carolyn Dompsey-Foss, Lee Beaumont. Program Evaluation 'n*' 
Extonaiurr A Comprohonsivo Study of Methods. Practices and Proco* 
duras Morgantown W Va Olfiee ol Research and ^Development, West 
Virginia University 1981 pp 19 20 
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Directors of state extension services are asking county 
extension agents for well-substantiated reports on the 
accomplishments of extension programs in order to 
provide state and federal funding sources with examples 
of extension's irtipact In connection with a symposium 
on the evaluation of extension programs, 2 approximately 
25 individuals directly responsible for formal evaluation of 
extension programs commented on federal, state, and 
county evaluation needs. By and large, their comments 
implied a need for county extension staff competency in 
program evaluation. A few of their comments follow, 

In regard to the three)evels of evaluation needs, I believe 
the most pressing neeq rests with the staff at the local 
program level, , . 

We specifically need evaluation for accounting to county 
and state government. The form and content may vary 
between what is needed by county commissioners and by 
* state legislators, , + , 

Can the meeting of needs for evaluation at the county 
level be structured in a way that will help to meet needs 
at the state and federal level as well? If so, how? 

Are actors at all levels of government asking questions? 
Do we have to evaluate for all the various actors every 
time we evaluate? 

How can we enhance coordination at all three levels? 

There^s an unquestionable need for separate but coor- 
dinated studies of program results at the national state, 
and county levels. National studies draw evidence from 
' several and sometimes all states: state studies draw 
evidence from several and sometimes all the counties in a 
state but county studies draw evidence from, and may be 
generalizabie to, a single county. State studies of exten- 
sion program results can be used to exemplify national 
program results, and county stupes to exemplify state 
and national results. 



Who Should Conduct Studies of 
County Extension Programs? 

m Should or can county extension staff conduct studies 
of the fesults of programs in individual counties? 
Regarding who should participate in extension program 
evaluations, below are some relevant questions raised by 
some of the attendees of the symposium cited above, 

How should we approach evaluation at the local level? 
Has our motivation of local staff to evaluate been handled 
properly? How can staff really be helped? 

What type of entry -level background is needed for exten- 
sion agents to be able to conduct program evaluation? 

Local extension staff are educators with the responsibility 
for delivering subj&t matter in some specific area of 
expertise, As educators, they ai\ expected to have the 
competencies that professional educators possess— the 
ability to design, deliver, and evaluate programs. Without 
all three competencies they are not effective educators 
and in reality operate intuitively rather than protussionally 

The local extension staff definitely needs training and 
assistance in evaluating local programs, ♦ ♦ * 

To what extent should local-level staff do program evalua- 
tion versus state specialists and or state level evaluation 
teams? 

Evaluation specialists seldom are avaijable to conduct 
studies of program results in single counties or even 
groups of counties. And if coynty staff in a state annually 
conduct dozens of studies primarily for county decision 
makers, state evaluation specialists will be unable to 
provide much assistance to each study. Thus, county 
extension staff must take major responstbilitytfor meeting 
identified county needs for documented studies of 
program results, 

In a number of state extension services, county staff 
are urged to collect, analyze, an£N{Tterpret evidence in 
order to evaluate the results of extension programs, But 
staff frequently reply: "First, show us how to get credijUd 
evidence of program results without spending an incrdfl^ 
ible amount of time doing It!" 

Mqst county extension staff do not have sufficient time 
or training to engage in rigorous evaluative research; they 
want simple and acceptable procedures for documenting 
the outcome of their work They want information on the 
results of their programs that is of sufficient value to 
justify the work required to obtain it, in other words, they 
want information that will improve their accountability, 
their programs, their understanding of the programming 
process, and/or their morale and satisfaction ^ 



*M F Smith fed.). Curraoi i$aU93 Probhma in Evaluating Cooperativa 
Eutamton Programs; A Symposium Gain&svilto. Fla Cooperative 
Extension Service University of Florida, 1981 pp 97-99 



RAP: A Feasible Approach 
County Extension Staff Can Use 
to Study Program Results 

Compared with the expectations of extension adminis- 
trators and their own expectations, county staff seem to 
obtain too little systematic evidence on program results 
Limited time and insufficient training are among the 
reasons systematic evidence is not obtained: another 
reason is that, until now, there was little in the way of a 
practical, general method county staff could use to 
conduct such studies. 

Using RAPs simple and acceptable method, county 
extension agents can now systematically document 
program results, and they don't have to spend an undue 
amount of time doing it For example, an agent can 
obtain systematic evidence on the results of a one- to 
three-year extension beef-breeding program, Initial 
training generally would take about one day; conducting 
the study generally would take five or six agent workdays. 

What is a "credible" amount of time that an agent 
should spend studying the results of a program in a 
county? Let's assume that an extension program requires 
300 agent-hours per year (seven and one-half weeks). 
Then 45 hours would amount to 15 percent (45/300) of 
the time expended on the program Furthermore, if RAP 
is used to study such a program conducted over three 
years, then the study would take only 5 percent of the 
total programming time Finally, keep in mind that 45 
hours is only about 2 percent of an agent s total annual 
programming time, assuming 1 ,800 hours are spent on 
programming per staff year, 




Features of RAP that Encourage 
Its Use by County Extension - 
Staff 

1* RAP is a simple, nonthteat&ning procedure for stuffy? 
ing program results. RAP helps agents get a quick start at 4 
studying results of one of their programs RAP empha- 
sizes agent skills in group process rather than technical 
methods of program assessment RAP standardizes and 
simplifies acceptable program evaluation methods for 
those with limited experience in formal evaluation 

2, BAP provides steps for studying the results of practi 
callymy extension program* With RAP's standardize' 



interview items ; agents simply 'plug W the educational 
methods subject matters, and expected end results 6f the 
program being evaluated. This innovative, easy-to-u3e, 
^fiil-in-the-bianks 1 method fehabfes county staff, with 
minimal assistance, to evaluate the effectiveness of their 
programs, Specifically , RAP Is a standardized application 
of a levels-ot-evidence model for evaluating the results of 
extension programs. 31 . 

RAP s interview ileitis cover a broad range of events 
and consequences; rather than a narrow and detailed 
scope. The general nature of RAP s Interview items allows 
a broad base of mater jal to be covered within thet 
confines of a brief interview., 

3 RAP applies to programs aimed at developing indi- 
viduals and groups as well as programs aimed at 
improving economic and social conditions, included in a 
program on building rnanagemtfft decision making 
among farmers and homemakers mighty be program 
content on evaluating and utilizing information: in a 
program for 4-H youth on building strong interpersonal 



relationships, generalizations about how hurnans 



m a 



things and form attitudes might be included 
program for community leaders olNqeeting Community 
needs, procedures for Idenfifyiqg and assessing these 
needs might be presented , TM RAP items ^an be modi- , 
lied for any of these kinds of content areas ,\hey also » 
can be adapted equally well to tho educational content of 
programs to improve economic and social conditions 
Such programs might include facts on fertilizers, animal 
and human nutrition, or water supply technologies. 



'ttiaude F Bennett Att&lyW*Q tffl&$ti§ 0/ f.Wctftd'Ort &r&g'r£m&.. 
ESG-57S., £* tension Servfct^USDA. Qmte * &$fiM\\ anU^ 
Donald I .Nelson., Atmlyimj m&KtS 0t CQfiifftvMy XXtVviopmvjii , 
Mississippi State., Miss ^Southorn flurai OOvtMoPment Center. Missis- 
sippi Slate University. 1975 
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4, RAP provides an 'on-ffteVob' method for evaluating 
program effectiveness, RAP provides more valid evidence 
of program results than casual observation would provide 
In terms of Exhibit 1, RAP helps field staff advance from 
"self-checking evaluations" to studies done "on the job/' 

Exhibit 1 

Degree of Complexity and Validity in 
Program Evaluation Procedures 



Snap 
Judgments 


Self- 
Checking 


On-the-Job 
Studies 


Special- 
Assignment 
Studies 



Lesser 



Greater 



As important as self-checking evahjations are, they can 
be very misleading. They are subjective appraisals by the 
provider of a program ^rid as such can contain unaccept- 
able error due to unconscious bias. 4 t 

RAP studies may not rival those by people who have 
special part- or full-time assignments conducting studies, 
RAP studies are, however, as complex and valid as "on- 
the-job'* studies by qounty staff can be expected to be. 
By taking an active role in studying how clientele view the 
effectiveness of extension programs, county staff gain 
firsthand insight into how to improve both their programs 
and their methods of reporting to resource allocation and 
support groups. - 



RAP encourages program personnel to participate in 
the collection and analysis of data, rather than rely on 
independent ev^lu'ators. Extension ^aff who implemented 
the prograrrrunder study should do part of the inter- 
viewing and participate in the analysis and interpretation 
of the data. Firsthand discovery can enhance agents', 
acceptance of the study *s findings and can motivate them 
to act upon such findings by presenting them with 
evidence from a cross section of the program audience. 

RAP exemplifies a trend toward inclusion of honre- 
searchers in evaluations of educational programs. Windle 
suggests that as part of the natural history of a profession 
some of the skills become routine so that less specialized 
groups can perform particular roles in the profession. 5 
RAP is designed to assist those who have not specialized 
in program evaluation to perform a significant role in the 
formal evaluation process. With RAP, more counties can 
systematically evaluate their programs than is possible 
when they depend upon program evaluation specialists. 

5. RAP encourages field staff supervisors, state 
program specialists, and volunteer leaders to help collect 
and use the evidence obtained from the study. Participa- 
tion in RAP by volunteer leaders and state and t district^ 
extension staff has^everal advantages: 

• People who conduct a study generally are more 
disposed to believing and using the findings than those 
who receive«only a report of the findings, 

• Whereas a county agent may invest &s much as 45 
hours in. a RAP study, 135 hours might be required to 
plan and implement the study. Thus, a team approach 
generally is most feasible. 

If one assumes that extension has a responsibility to 
[help build community and business leaders' abilrty to 
I systematically evaluate the effectiveness of public 
>CQarams, then by encduraging volunteer leaders to 
participate in the study, extension is fulfilling that goal. 
Lay (volunteer) committee members, county extension 
chairpersons (in casps^here they do not lead the RAP 
study), district agents, and specialists are encouraged to 
conduct half or more of the RAP interviews. The county 
staff member leading the study conducts the remainder of 
the interviews under such an arrangement. 



♦This may be illustrated by the following example, ' In a program of 
teaching speech to deaf children, teachers initially found the children's 
speech unintelligible. After teaching for six months, the teachers felt that 
the children's ability to make themselves understood had increased 
dramatically However, data collected during the course of the training 
indicated otherwise, Tape recordings of the children's speech were 
taken at regular intervals during their training Impartial observerif who 
listened to the recordings could not distinguish when given recordings 
by the, children were taken— early in their training or late in their 
training, What apparently happened is that the teachers had learned the 
linguistic code of the children; the teachers had changed—not the 
children." See John M Gottman and Robert E. Clasen, Evaluation in 
Education Itasca. HI,: F. B Peacock Publishers, Inc., 1972, p. 2 



-Charles Windle. Trends and Problems in Non-Researcher Participation 
in Program Evajuation/' presented at the annual meetiha, of the Evalua- 
tion Research Society, Nov 19-22, 1980, Arlington, Va 




13 



Validity of Reflective Evidence 
for County Users of Evaluation 

RAP depends on reflective evidence, so-called because 
the interview procedure requires program participants to 
reconstruct (reflect upon) their feelings, behavior, and 
condition before, during, and following their participation 
in the program being studied. Interviewees estimate the 
amount of chahge they experienced or observed that can 
be attributed to participation in the program. This 
perceived "before and after" evidence of program 
effectiveness— "reflective" evidence—is one way to deal 
with the attribution problem, namely, to what causes or 
influences a change is attributed. For example, in an area 
where farmers increased their production, what demon- 
strates that extension had a part in bringing about the 
change? 6 

Some social scientists, including some program evalua- 
tors, maintain that the only way to obtain adequate , 
evidence of program results is by observing what clientele 
actually do and receive as a result of program participa- 
tion/Such analysts contend that: (a) what clientele 
perceive, believe, and say are the results of their partici- 
pation in a program is invalidated by what they want to 
believe in order to feel good about themselves and their 
past actions; and (b) that reflective evidence is invalidated 
by memory loss or distortion. 7 People who accept this 
position are object ivists, for they favor using a natural or 
physical science model for evaluation studies. 8 Object!- 
vists rely on rigorous study designs to exclude or take 
into account other causes of clientele change besides 
extension. 

Analysts who maintain an interpr^N^pr subjectivist 
position 1 emphasize that human experience is perception 
and that perception should thus be a focus of study. 



'Claude F. Bennett, Analyzing impacts of Extension Programs, 
ESC-575, Extension Service-USDA, 1976, pp. 15-20. 

'See, for example, Peter H Rossi and Sonia R. Wright. "Evaluation 
Research/' Evaluation Quarterly, Feb. 1977: also, Mary Ann Scheirer. 
"Program Participants' Positive Perception: Psychological Conflict of 
Interest in Social Program Evaluation/" Evaluation Quarterly. Feb 1978 ^ 

•See Ernest R House, Evaluating with Validity, Beverly Hills, Calif.: Sage 
Publications. 1980 

•Laverne B. Forest and Mary G. Marshall. Impact of Extension in . 
Shawano County. Methodology. Madison, Wise.: Program and Staff 
Development, University of Wisconsin-Extension. 1978 
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*Such analysts believe that it is both necessary and gener- 
ally more feasible to obtain evidence on what clientele 
say they perceive to be the results of program participa- 
tion. Subjectivists maintain that it is necessary to obtain, 
the meaning of a program to its participants. For 
example, analysts who use perceptions to study program 
results maintain that: 9 

•^Perceptions allow respondents to interconnect events 
and to identify the cumulative effects of multi-year, multi- 
method programs. 

• Perceptual data are more easily understood by study 
users who may not understand how numbers of changes 
in people or institutions indicate the value of a program. 

The intent here is not to contribute to the objectivist- 
subjectivist debate, but rather to assert that subjectivist 
(e.g., reflective) evidence is appropriate for county studies 
of exte nsion programs for many reasons, including: y 

• Reflective evidence can be collected from progr> 
participants after their participation r^(Rer4tei^bot 
before and after and from both participants and nonpar- 
ticipants (comparison or control groups). * 

• RAP's "plosed-end" (multiple-choice) interview items 
permit many possible specific answers to be recorded 
and aggregated within a few general response categories. 
For example, if a question asks the extent to which 
program participants implemented skills learned from an 
extension program, two respondents mighf both say that 
they put to use "to a great extent" the ideas or skills they 
learned about prevention of hoffie burglary. One partici- 
pant may tyave installed a superior door lock; the other 
may have inscribed identification nuVnbers on his or her 
possessions. 

• Reflective evidence generally will be acceptable to the 
principal users of the findings—agents themselves, 
campus staff, volunteer committee mefnbers, county 
legislators or commissioners, state legislators, and others 
who are "close to the extension programming process." 
Such people know enough abput local extension 
programs to assess for themselves the validity of the find- 
ings and to interpret the findings in the context of their 
existing knowledge. 

• The information on program effectiveness that is 
gleaned from a RAP study is far more complete and valid 
than that found in most county-level extension reports. 

Summaries of RAP studies can be included in routine 
channels for internal accountability and in regular means 
for reporting to county and state funding bodies and the 
public. 
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Reducing and Minimizing 
Potential Problems 

Like any technique, RAP has limitations and potential 
problems; these problems can, however, be reduced and 
kept to an acceptable minimum. Five potential problems 
are identified below, along with suggestions on how to 
reduce of minimize them. 

1. Vested interests of extension staff will bias the study. 
Bias generally can be minimized if the study is conducted 
by a team representing county, district, and state exten- 
sion staff and volunteer extension leaders. Each of these 
groups has different biases and vested interests, so biases 
will tend to cancel each other out. The recording of 
open-ended responses is particularly subject to inter- 
viewer bias; .if several interviewers conduct the interviews, 
however, a more valid pattern of responses should be 
obtained. 

It is most important that all those involved in a RAP 
study be aware of the self-defeating effects of bias. RAP 
studies will lose credibility if their audiehces find bias in 
them. 

Finally; competency in the use of the RAP approach is 
perhaps the best guard against bias. 

2. Lack of evaluation expertise by extension program 
staff can reduce the accuracy, completeness, and useful- 
ness of the study's findings. Trial RAP studies in Ohio 
and New York State indicated that extension staff teamed 
with lay leaders could do accurate and useful studies. As 
extension becomes more familiar with RAP, there will be 
an increasing number of exemplary RAP studies and 
increasing numbers of people experienced with the 
process* . 

3. RAP's standardized interview items may elicit resultp 
that are too general, vague, or incomplete. If 'the evidence 
from a RAP study is too sjbnerajor spotty to adequately 
cover the results of the program feeing evaluated, then 
perhaps the scope of the study vws too wide or not well 
adapted to the particular si taation . RAP's probe items will 
detect with some precision examples of what people did 
aqd/or received as a result of their participation in the 
program being evaluated. It is the responsibility of the 
RAP team to ensure that areas that are not sufficiently . 
addressed in the standardized items are covered in addi- 

9 tional locally developed interview items. 

7 The staff implementing a RAP study may need assist- 
ance from a program evaluation specialist to ensure that 
the scope of the study is manageable. Such specialists 
also may be helpful in constructing or adapting interview 
items directed at issues that are not sufficiently addressed 
by the standardized items. 

4. Reflective evidence on the results of a program may t 
be based more on the interviewees' overall attitude 
toward extension and its personnel than on their percep- 
tion of the actual results of the program being studied. 
Peopl^who feel positive toward something usually hesi- 



tate to admit to themselves or to anyone else that the 
results of that Something (in this case a particular exten- 
sion program) were not positive or beneficial. This ^ 
tendency can be minimized, however, by judicious use of 
the open-ended (probe) follow-up questions suggested in 
the RAP guidebook. 

If a significant portion of the respondents are unable to 
provide specific examples to support their general 
assessment of the results of their participation in a 
program, then the interpretation of the study's findings 
should reflect this. Furthermore, responses to the stand- 
ardized items should be interpreted in light of their 
consistency or inconsistency "with the open-ended 
responses. 

A study of a program with a relatively limited scope is . 
more likely to adequately elicit interviewees' perceptions 
about the results of a program than a study with a 
broader scope. 

5. Reflective evidence may be invalid or not sufficiently 
complete if program participants have forgotten the effec- 
tiveness of the program. People frequently forget where 
or from whom they learned some^ing. Thus, for a RAP 
study ro be most valid (a) less than two years should 
have elapsed since the interviewees participated in the 
program, and (b) the program should have been distinct 
and forceful. Several additional techniques can minimize 
the effect of memory loss (a) Adequately but succinctly 
describe the program to interviewees to refresh their 
memories of the program and their participation in it; (b) 
send a copy of the interview instrument to interviewees 
for review before the interview; and finally, (c) use the 
open-fcnded questions to determine whether an inter- 
viewee has a solid basis for his or her estimates of 
- changes resulting from the program. 



Summary 

The Reflective Appraisal of Programs (RAP) approach 
provides county extension staff with a simple method for 
obtaining clientele-perceived assessments of the results 
of extension programs. Furthermore, the method is 
consistent with the readiness, strengths, and limitations of 
extension agents to conduct such studies. 

The "RAP package" includes the following: 

• Guidelines for selection of the program most in need of 
systematic study. 

• Levels of evidence that can be used to describe and 
evaluate clientele-perceived results. 

• Standardized but modifiable questions for eliciting . 
information from participants regarding results of the 
program in which they were involved. 

• Guidelines for analysis, interpretation, and utilization of 
the study's findings. 

Exhibit 2 on page 9 summarizes the advantages of 
RAP and how to minimize potential problems. 



Appendix: The Role of RAP in 
In-Service Education for 
Extension Staff 

Hundreds of articles and books have been published 
over the past 10 years on program evaluation, and 
dozens on extension program evaluation. Specific, stan- 
dardized procedures, however, rarely are included in such 
publications. I developed RAP because I found other 
approaches to evaluating program effectiveness could not 
be taught in brief workshops on extension program 
evaluation. Proof of RAP'S success was quickly evident. 
After I began to use RAP's standardized approach, partic- 
ipants' ratings of workshops where I presented the 
approach rose dramatically. 

RAP grew- out of two professional experiences, one 
lasting over several years; the other, several months. The 
first experience— a frustrating one (a feeling shared 
perhaps by other program evaluation workshop 
instructors)— involved attempting to teach field staff 
within one day how to study program results. (Invitations 
from state extension services to provide such in-service 
education rarely were for more than one day.) The 
second experience — a successful one— involved helping 
an individual area extension staff member conduct a 
study of program results. 

I tried two approaches at the one-day workshops, but 
found both unworkable. First, I presented optional ways 
of studying program results, thereby helping the trainees 
select from many alternative methods for obtaining 
evidence to evaluate their programs. Advantages and 
disadvantages of various methods of obtaining data- 
interviewing, mail questionnaires, observations— were 
presented (see Exhibit 3). Within the confines of the brief 
training session, however, most county staff were unable 
to develop an evaluation plan, given the "mind-boggling" 
number of possible methods of <a) categorizing program 
results, (b) involving people ih evaluation procedures, and 
(c) measuring program results. r 

Providing workshop trainees with a variety of sample 
questionnaires, observation devices, and other instru- 
ments that have been used in evaluation studies did not 
help much either. As the participants tried to figure out a 
method to measure the results of their own work, they 
realized that few, if any, of the instruments fit their 
specific needs. 

The second approach I tried, also in brief workshop 
situations, was to lead field staff trainees through a 
sample evaluation study. This case study approach had 
two advantages: (1) It enhanced communication during 
the workshop as the study and findings were discussed, 
and (2) it simplified the content of the workshop by 
reducing the number of evaluation methods being pre- 
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sented. But, by focusing attention on one study, however 
exemplary the study may have been, I tended to lose the 
interest of the majority of the staff participating in the 
training. Extension staff have extremely diverse interests; 
they deal with vastly different audiences and subject 
matters. Within the same county, for example, some 
agents are interested in solving housing problems and 
clothing problems, while others are involved in solving 
livestock and grain problems. Within a single program 
area such as 4-H, programs may be vastly different, too, 
even within the same county. Staff are interested in 
finding out how to better document the results of their 
own programs, not in how someone else obtained such 
documentation. 

The critical question seems to be this: How can the 
program evaluation methods discussed in a workshop be 
sufficiently specific to be easy to explain and easy to use, 
yet applicable to the extremely broad range of interests of 
extension staff? One answer is RAP. Its innovative, stand- 
ardized interview questions allow program staff to "plug" 
practically any subject matter and educational method 
into its items. RAP is a specific technique for evaluating a 
program, yet it can be adapted to studies of practically 
any extension program, subprogram, or phase of a 
program. 

The second experience that led to the development of 
RAP was my work with a former extension agent in Okla- 
homa, with whom I conducted a study on the results of a 
community development program. 10 This study demon- 
strated in detail how the levels-of-evidence model, 11 , 12 
could be applied to an evaluation of a multi-county 
program. RAP is a modification of the methods and inter- 
view items we used in that study. Once the references to 
the specific content and/or processes of the Oklahoma 
program were removed from the interview' items, we were 
left with the first approximation of the "content-free" 
items that are introduced in RAP. 4 

By presenting prestructured items within an overall 
planning, implementation, and utilization strategy, RAP 
makes it possible for county staff to produce well- 
documented, comprehensive evaluations. 



,0 Claude F. Bennett and Marvin D, Oldham, Extension's Role in a 
Concerted Effort Toward Rural Development: Analysis and Evaluation. 
Stillwater, Okla. Cooperative Extension Service, Oklahoma State 
University, 1976 

"Claude F Bennett and Donald H Nelson, Analyzing Impacts of 
Community Development Mississippi State, Miss,: Southern Develop- 
ment Center, Mississippi State University, 1975. 

1 'Claude F Bennett, Analzying Impacts of Extension Programs, ESC- 
575, Extension Service-USDA. 1976. 
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Exhibit 2 

Summary of Strengths of RAP, with Corresponding Potential Problems and 
Suggestions for Minimizing Problems 



Advantages of RAP 



Potential Problems 



How to Guard Against 



A. Organizationally Relevant 

1 Serves primarily county-level 
decision makers. 



2* Planning aids &nd standardized 
items for interviews help reduce 
time needed to study extension 
program results, 



RAP will not suffice in providing the 
information on program results 
needed by state-level decision 
makers. 

Staff who implement RAP may not 
grasp the logic behind 
program-evaluation study 
procedures. RAP users may fail 
to realize that RAP is only one 
of many program-evaluation 
methods. 



Formulate complementary plans for 
meeting state-level decision makers' 
information needs! 



Encourage staff to use RAP as an 
initial method to study program 
results but to employ other 
methods as their skills grow. 
Offer opportunities to learn 
about program-evaluation 
methods other than RAP. 



B. Methodologically Relevant 

1, Volunteer leader participation in 
studies of program results 
ensures study legitimation, utili- 
zation, and assistance. 

2. RAP's standardized approach to 
program description and inter- 
view items covers most essen- 
tials while saving staff time, 



3, Reflective evidence on program 
results is easier to obtain thdn 
evidence obtained both before 
and after a program, requires 
minimal time, and permits esti- 
mates of results of combinM 
educational methods/content, 

4. RAP is a "dq-it-yourseir 
method that does not require 
resources/guidance/control of 
the study by evaluation special- 
ists or social scientists. 



Reduced control of interviewing 
procedures, analysis, and confiden- 
tiality. 

RAP evidence may be too vague to 
be meaningful. Interviews may not 
elicit evidence on program nuances, 
critical incidents^ and some types of 
program results. 

Interviewees' estimates of program 
results may be affected by their atti- 
tudes toward program personnel or 
extension or by limited recall ability. 



Program staff may view RAP as a 
means of obtaining only positive 
evidence of program results for public 
relations purposes and will therefore 
tend to bias the study findings. 



Select participating lay leaders care- 
fully according to education, expe- 
rience, and openness to training 
needed for their role in RAP. 

Reduce the scope of RAP studies. 
Encourage RAP users to adapt and 
supplement standardized procedures 
as necessary. 



Rely on probe questions to discount 
interviewee responses that seem to 
have no factual basis, Encourage 
"don't know/don't recall" responses 
when applicable, 



Train competent teams, including 
volunteer leaders, to implement RAP, 
Caution that RAP studies will lose 
credibility if audiences detect unac- 
ceptable biases. Emphasize the value 
of negative findings for making 
program improvements. Reward staff 
for basing program improvements on 
RAP studies. 
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Exhibit 3 

Alternative Methods for Collecting Evidence on Program Results 



Methods 



Interviews: series of oral questions 



Questionnaires: series of written ques- 
tions answered and then returned 
through the mail 



Expert Opinion: judgment based oh 
people's experience and competence 
in a particular study 



Observation: organized surveillance 
and analysis of behavior 



Analysis of Documents: analysis of 
official papers that constitute the 
written records of extension program 
administration, including newspaper 
clippings, farm and home records, 
4-H records, etc. 



Advantages 
personal contact 

flexible; permits follow-up questions 

provides opportunity for expression 

without fear of embarrassment 
< 

low cost per respondent 

many tangible and intangible factors 
can be taken into account 



eyewitness account / 

allows comparison of words and 
deeds 

low cost 

source of background information 



Limitations 

high cost per respondent 
time consuming 

inflexible; discourages follow-up 
questions , 

low response rate w 

human error and bias 

difficult to check information used to 
reach conclusions 

human error and bias 

high cost; time consuming 

inapplicability of information 
"selective survival" of documents 
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Preface 



Extension agents have already used the RAP approach 
* to study the results of these and other programs:' 



Reflective Appraisal of Programs (RAP) for studying 
results of extension programs is a simple and sufficiently 
valid method extension staff can use to document 
clientele-perceived results of a program. The method may 
be used without expending an undue amount of time in 
documenting such results. 

This publication is intended to be an instructional guide 
for county extension staff who wish to use RAP to deter- 
mine and appraise (evaluate) results of their programs. A 
companion publication in the "RAP package" presents 
the rationale for RAP and compares it with other 
approaches used to determine extension program results. 

County extension staff often obtain clientele s imme- 
diate reactions to individual extension activities. These 
reactions help agents to gauge immediate results of 
events within k program and to plan subsequent events. 
This guide is based on the premise that county extension 
staff should also ascertain the longer-term results that 
occur over the months or years following a program, The 
audience for these findings will include extension agents, 
and also local, county, district, and state persons who 
influence or make decisions on program direction and 
resources 

This guide provides both the background concepts and 
step-by-step instructions extension agents need ih order 
to determine program results A workbook accompanies 
this guide as the final publication of the three-piece "RAP 
package/' The planning aids in the workbook can help in 
choosing and recording specific plans for each step of a 
RAP study. 



• Swine disease control program 

• Community recreation development program 

• 4-H environmental education program 

• Summer telephone advisory program on fruit and 

vegetable preservation 

• Homemaker club program 

• Integrated pest management program 

• Consumer education program 

• Teenage sex-education program 

• 4-H home fire-prevention program 

• Handicraft marketing program 

RAP is based on a levels-of-evidence model for classi- 
fying the results of extension programs. These levels of 
evidence are fully described in the following sources: "Up 
the Hier&rchy," Journal of Extension, March/April 1975; 
Analyzing Impacts of Community Development, Southern 
Rural Development Center, Mississippi State University, 
1975; Analyzing Impacts of Extension Program, ESC 575, 
Extension Service, USDA, 1976 (reissued in 1979); 
Teaching Materials on 'Seven Levels of Evidence": A 
Guide for Extension Workers, Supplement 1 to ESC 575, 
Science and Education Administration, USDA, August 
1980. 



/ 



The first county extension agent to complete a HAP study was Larry C 
Auit of Richland County Ohio His study was entitled Using Rellectlve 
Evidence of Farmers to Evaluate the Richland County Integrated Pest 
Management Program' (1980). 
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Step 1: Gauging Your Interest in 
the Results of Extension 
Programs , 

Who Needs What Evidence on the Results 
of Programs? 

If you are like most extension staff, you have done a lot 
of work in the past year or two. You have probably held 
hundreds of meetings, answered many people s ques- 
tions, and sent mailings by the thousands. You may have 
conducted TV or radio shows, also. But can you satisfac- 
torily answer this double-barreled question: Who needs 
what evidence on the results of your programs? Are you 
jand others getting enough evidence on the results of 
your programs, or is there evidence on ttte results of 
programs that you and others really need but do not 
have? 

Do People Get Their Money's Worth? 

Do you know enough about the results of people s 
involvement in extension activities to account adequately 
to others— county commissioners, extension advisory 
committees, your district director? They each have a 
responsibility to judge whether extension programs are 
worth their costs. 

Are the Programs Worth the Effort? 

Do you know enough about the results of people s 
participation in programs to answer this question; Is the 
payoff of the program worth my time and the time of 
people who work with me— paid staff and volunteers? 

How Can Programs Be Improved if You 
Don't Know Their Results? 

Finally, do you and others know enough about the 
results of your programs to see how to help people more 
effectively? In other words, do you know enough about 
the results of programs to see how to improve them? 

Better information on the results of past programs can 
help you improve future extension programs—their objec- 
tives, methods, and financial support. 

Many extension workers get bits and pieces of informa- 
tion (evidence) about the results of programs, In their 
attempts to combine these bits and pieces of information ( 
they often end up creating program evaluation reports 
that are stranger than the proverbial camel: "a horse put 
together by a committee/' 

If you are interested In learning how to systematically 
document extension's effects on program participants, 
then this publication is meant fer^gu^Read on and we 
will show you how to do just tmKv^> 



Below is a shortened version of Planning Aid A. Plan- 
ning aids are only referred to in this guide, not 
completely presented. Planning Aid A and the other plan- 
ning aids— B through T— are fully presented in the 
accompanying RAP workbook. The planning aids are 
intended to help you choose options for implementing a 
RAP study, and to help you record these options. A 
completed workbook can serve as the outline for a brief 
written plan or proposal for a RAP study. 



Planning Aid A: Choose a program for a RAP 
study. (See page 1 of the RAP workbook.) 



Step 2: Selecting a Program for 
a RAP Study 

Three Factors to Consider Before 
Selecting a Program for Study 

Before you can select a program for your RAP study, 
you must first evaluate which program has the highest 
priority for such a study. The three factors shown in 
Figure 1 should help you do this (based on Harry P. 
Hatry, Richard Winnie, and Donald M Fisk, Practical 
Program Evaluation for State and Local Government 
Officials, Washington, D The Urb^n Institute, 1973, 
pp. 108-112.) 

Figure 1 

Use Three Factors to Help You 
Select a Program to Study 



Factors 
1; Need 

2 EfiaaMiiy 

3 Persuasiveness 



County Fair 

Volunteer 

Leader 

Training 

4-H Interest 
Groups 



4 



4-H Community 
Development 

4-H Camping 
Programs 

4-H TV 

4-H Livestock 
Club Work 
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Factor 1: Which program Is most In need of a study of 
Its results? Have your supervisors or groups external to 
extension requested information regarding the results of 
specific programs? Evaluation studies are often mandated 
for pilot projects, new programs, and Continuing 
programs as extension, its funding sources, or its clien- 
tele raise questions about the results of these programs. 
If you have been asked to account for the results of a 
particular program, the decision as to which program to 
select for a RAP study has been made for you. 

If you have latitude regarding which program to eval- 
uate, there are three aspects of "need" that you should 
consider. First, how certain are you and others about the 
effectiveness of each program? You are likely to feel less 
certain about the effectiveness of pilot or new programs, 
mass-media programs, or programs that have not been 
evaluated as recently as others. If the program is large or 
highly visible, the need for evaliAtion is heightened even 
further. . 

Second, how much will you and others gain if you can 
show thgt the progr^nris working well? There might be 
an opportunity to increase the size of the program if you 
can document its results, pr you might be able to 
increase the budget by 'producing such documentation. 
For example, the following types of programs have a high 
potential for increased or permanent funding if positive 
results can be demonstrated: 

• Legislatively, administratively, or privately funded 
special projects 

• Programs of high interest to legislative groups or the 
general public 

• Programs with insufficient resources to permit partici- 
pation by all who have expressed interest. 

Third, how much risk are you and others taking if a 
program is not working well? For example, some 
programs may have declining participation or may require 
resources that seem too large for the apparent benefits, 
In other programs, participants could be harmed if they 
did not apply or If they misapplied information conveyed 
to them, 



Factor 2: Which program can be studied most feasibly? 

Some programs are more "evaluable" than others. 
Programs differ in their clarity of objectives, There may 
be financial barriers or political barriers to obtaining 
evidence on the results of some programs. The following 
specific questions should help you determine how conve- 
nient and practical it would be to evaluate the results of a 
program you are considering for a RAP study. 

0 

• Is it relatively easy to distinguish the program from 
other programs? 

• Does the program lend itself to collection of data from 
clientele? 

• Does the program have cidaf criteria by whioh it can be 
evaluated? N 

• Can the program be studied without disrupting it? 

Factor 3: Which program Is most likely to be modified If 
a study shows need for modification? How persuasive are 
your findings likely to be? For example, studies of 
program results will have more influence on decision 
makers if; 

/ 

• vested interests in maintaining the program as it Is are 
not as strong as they are in other programs, 

• preconceptions regarding we program's effectiveness 
are not crystalized. 

• the program and administrative staff have several 
options with regard to the program, 

• the program staff want to continue the program but 
have noticed problems in delivery or clientele response. 

In addition to the considerations above, programs for 
RAP studies can be Selected within the context of an 
established process such lis constructing an annual plan 
of work, multi-year plans, county reviews, etc 



• Participants could be harmed handling dangerous 
chemicals while attempting to apply Information 
conveyed by an e^ension program. 

• Participants could be harmed if they had nutritional 
problems that were pot taken into consideration during a 
program. 

• Participants could be harmed if they misapplied food 
preservation techniques they were taught during a 
program 

• Participants could misapply crop production techniques 
promoted by a program 
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Step 3: Identifying Who Will Use 
and implement the RAP Study 

Proving a RAP study is orte thing, but getting it , * 
accepted and used by others is quite another. Michael 
Patton, author of Utilization-Focused Evaluation, has 
commented: "You can lead a decision maker to informa- 
tion, but you can't make him swallow it." 

What can you do to ensure that your study will influ- 
ence people who can help you make needed program 
changes and supply propram resources? To answer this 
question, let's first briefly review the reasons for obtaining 
systematic information on the results of extension 
progryns. 

1* To improve extension's programs 

• by improving program staff decisions 

• by improving council or program-building committee 
decisions . 

• by improving administrative decisions 

2, To improve extension's accountability 

• to county, state, federal, and private funding sources 

• to the general public 

• to extension and untversity*administrators , m 

• to lay committees and support groups 

3, To jmgfove understandings! and communication 
about extension programs 

•~ by clarifying program objectives 

• by analyzing and describing the costs, processes, and 
outcomes of programs 

4, To improve morale 

• of productive extension staff members 

• of program participants who have made progress »■ 

Try to pinpoint the reasons different people or groups 
might have for studying a program's results. Who might 
use evidence on the results of programs, and how would 
they use it? For example, potential users of your RAP 
study might have the following types of decisions to 
make: 

• whether to recommend or approve the same, more, 
fewer, or no resources for a program 

• whether to revise the objectives of a program 

• whether to modify the educational methodology and 
content of a program 

• whether to modify the intended audience for a program 

• whether to modify the delivery system of a program 

• whether to alter how a program is managed by 1 
revamping its organizational structure and procedures 

O 
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• whether to initiate or modify other programs, policies, 
and procedures related to the program being evaluated* 

Ensuring That Your Study Will Be Used 

Generally, tp&way-tq ensure maximum use of any 
study is tp^oftticlpate who^might have an interest in its 
tinding^and to involve the$9 people in the design, 
implementation, and interpretation of the study. H , 
Apply Seaman A. Knapp'sj maxim to get acceptance 
and use of your^tudy. 

"What a man hears 
he may doubt; y 

What he sees / * 
he may posaiWy ifoubt, 
But what he himself 
he cannot (S^oubt;' 

Seaman A. Knapp, 
forerunner, 
Cooperative Extension Service 



Planning Aid B; Identify who might use the RAP 
study and for what purposes (See page 2 of the 
RAP workbook ) _ 



Recruiting a R^VP Team 

Remember, sharing responsibility for the purpose, 
method, and completion ot your RAP study can increase 
its relev.ance*credlbility, and usefulness. A team can also 
make the w^oad lighter and the job more enjoyable, 
Think seriously about who you would like to invife to- 
work with you in designing, conducting, interpreting^ 
and then, we hope, using— a RAP study^ 

Figure 2 

Decide Who Should Be Involved In a RAP Study 
of Program Results 



Persons or Groups Who Could 
Help with the Evaluation 

1* District director 

2 Program specialist^ 

County director(s) 

4 



5 



Extension agents (in 
own and other counties) 

County councils or 

program-development 

committees 
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Planning Aid C Indicate the indivi0uals or 
groups you will invite to work with you, (See 
page 3 of the RAP workbook ) 



Step 4: Defining the Scope of 

the Study * 

t . * .. 

Distinguishing Between Implementation 
and Results . 

Objectives for extension programs exist at different 
levels (see Figure 3) The three lowest levels of 
objectives— the most immediate objectives for a* 
program— concern implementation of the program These 
levels are (1) extension staff invest a given amount of 
inputs (time and resources) in order to (2) conduct 
specified Ativities intended to obtain (3) people 
involvement in these activities The levels of objectives 
concerning the results of the program include (4) 
participants' immediate reactions to program activities: 
(5) participants' KASA changa^-knowledge, attitude, skill, 
and aspiration changes; (6) thwr practice change; and 
(7) the end results that occur as a consequence of the 
KASA change and practice change, 

Figure 3 

Lewis of Objectives in Extension Programs 



* Selecting the Activities and Program 
Participants You W}li Study 

The next step Is to identify the activities ^id program 
participants you will study. This will define thejscope of 
the evaluation, The activitieswid people involved in the 
< program should be defined as they relate to four traits: * 
v (1) the educational or delivery methods of the program; 
(2) the content of the program: (3) the audience for the 
program, and (4) the time period you will study (see 
Figure 4) 

Methods. The RAP approach is not intended for study 
of the results of a single event of* activity such as a work- 
shop. Rather, RAP helps you to examine the results of a 
combination of several delivery methods such as TV 
shows, newsletters, and demonstrations, 

In identifying the delivery methods that were used, be 
sure to include clientele-initiated activities For example, 
did clientele tend Jo ^all or visit the offte^jo request 
program-related fnfortnatiofi or advice? If they did, you Q 
should consider including these activities in the scope of 
your study Fwnned or unplanned ways of responding to 
participants /requests for information or advice is the *" 
most impotent educational method in some extension 
programs. 

Figure 4 

Use Four Program Traits to Help 
Define the Scope of a RAP Study 



6 



5, 



3 



2 



End Results 



Practice Change 



KASA Change 



Reactions 



People Involvement 



Activities 



Inputs 



Program Traits 
t. Methods 

2, Content 

3, Audience 

4, Time Period 



Consumer Economics 
Institute 
V 

Lectures by 
Resource People 

Television Spot 
Announcements 




Series of Newsletters 
on Money 
Management 

Budget Workshop 
for Young 
Homemakers 
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Content The content or subject matter of a program 
includes (1) psychological, economic, and/or social 
processes, or (J?) physical, chemical, and/or biological 
processes. Some extension programs focus exclusively 
on psychological, economic, and/or social processes, 
such as commo^Tfy^marketing programs. Others focus 
exclusively on physical, chemical, and/or biological 
processes, such as some pest management programs. 
And some programs include information on both (1) and 
(2) above, such as farm and home development 
programs. f 
// Audience. As you are defining the audience for your 
program, keep in mind that a program with two stages 
may also have two audiences. The following three exam- 
ples illustrate this. 

• Stage 1 of a p/ogram trains volunteer leaders of 4-H 
clubs and is followed by stage 2, in which leaders guide 
club programs with 4-H youth. 

• Stage 1 is a demonstration for farmers where they tr9* 
out a new practice. In stage 2, these farmers show neigh- 
boring farmers how the practice works. 

• Stage 1 of a program helps community leaders 
organize task forces to address community problems. In 

\ stage 2, the task forces begin their work. 

If you wish to evaluate the results of a program that 
has more than one stage, identify the scope of the first 
stage for one RAP study and the scope of the second 
stage for a separate, second RAP study. (You will need to 
complete two workbooks to do this.) 

time Period, People's memories tend to lapse increas- 
ingly as they are asked to recall events that took place a 
lopg time ago. For this reason, your study should eval- 
uate only events that took place less than three years 
ago. 

The scope of your study should be sufficiently general 
to be meaningful and important to the people who will 
potentially use it. In other words, it should: £ 

• be applicable to the entire program or its principal 
parts 

' • deal with a number of activities 

• deal with the main themes of the program. 

/ At the same time, the study should not encompass 
iuch breadth that it becomes unmanageable. If too many 
'audiences, subject matters, and methods are included in 
the scope of the study, it will become overly complex and 
unwieldy. 

Involve the users of the study in making the important 
decisions on the scope of the RAP study. t: 



Identifying the Results Expected from 
the Program 

For the purposes of this guidebook, expected results 
are considered as they relate to th& levels of reactions (to 
involvement in activities), KASA change, practice change, 
and end results. 

. Participants' reactions to program activities can be 
expected to vary depending on a combination of factors 
including the teaching Fnethods (e.g., confrontational 
versus consensus-seeking) used in the program, the 
subject matter (e.g., innovative versus conventional 
wisdom) of the program; and the clarity of the audience's 
standards (e.g., precise versus diffuse). 

Generally, you can expect that knowledge, attitudes, 
Skills, and aspiration (KASA) changes will closely relate 
Nip the program's subject matters. Likewise/any practice 
changes can be expected to closely relate to the subject 
matters. For example, a program with subject matter on 
wood burning In home heating would be expected to 
change an audience's knowledge, attitudes, skill^f inten- 
tions (aspirations), and practices regarding home heating. 

End results that can be expected from KASA change 
and practice change relate less closely to the subject * 
matters of the jpr.cjgrar^. Expected end results of the 
program on using wood as a source of hom£ heating 
may include, for example, increased comfort, savings in 
heating costs, and increased esteem in the community or 
neighborhood. 



Planning Aid D: Indicate the scope of the RAP 
study. (Sge page 3 of the workbook.) 
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Step 5: Identifying Interviewees 

A RAP study will be meaningful only if its findings 
include data from a cross section of the participants in 
the program being evaluated. A return rate of only 20-30 
percent is typical for mail questionnaires to most program 
audiences, and respondents tend to be those who are 
most positive toward the program and any who are 
extremely negative. We therefore recommend that you 
conduct telephone or personal interviews, in order to 
obtain responses from at least-75 percent of the 
program's participants. 

How many participants should you seek to interview? 
The answer will depend paYtly upon how many people 
participated in the program within the time period 
covered by the study. 

• If there were fewer than 40 program participants 
during the time period covered by the study, interview all 
the clientele the program reached during that period. 

• If more than 40 people participated during the time 
period, randomly select participants from mailing lists, 
attendance lists, membership lists, etc. 

Fortunately, interviews with 35-40 randomly selected 
participants yield information that is almost as accurate 
as that which would be obtained if all the participants 
were interviewed. If, however, you wish to obtain data on 
the results of the program as they are perceived by 
-'^iWerent groups, such as rural participants and urban 
participants, then you should Interview 35-40 rural partic- 
ipants and 35-40 urban participants. Likewise, if you wish 
to see whether the results vary depending upon the* 
delivery method that was used, interview 35-40 partici- 
pants who were exposed to one method, 35-40 who were 

* exposed to a second method, 35-40 who were exposed 
to both methods, and so on. 

' In addition, if the program includes more than one 
stage, such as a program to train community leaders who 
in turh lead committees working to improve various * 
aspects of a community, you may elect tainterview 

* participants from each stage. 




Ensuring That Your Sample Will Be 
Representative of the Participants 

You can best prepare for a* RAP study by keeping 
complete lists of the program participants while the 
program is being implemented. Using attendance lists, 
mailing lists, etc., compile a complete list of the program 
participants. Then follow these four steps to select a 
random sample. 

4 . Write in alphabetical order the names of the people 
who participated in the program during the specified time 
periocT and number this list. This list should include both 
participants who are still active and those who are inac- 
tive. 

2. Write a number— 1, 2, 3, 4, or 5— on each of five 
slips of paper so that no number is repeated and only 
one number appears on each slip. Place the five slips "in 
a hat" and draw one slip out. 

3. Circle the number and the name on the list that cor- 
respond to the number on the slip you have just drawn. 
Vou have just selected your first interviewee. 

4. Starting from the first name you have chosen, circle 
every second or third or fourth, etc., name after it so that 
40 names are chosen. For example, if you have 120 
names listed, you wouldchoose every third name. 

If you cannot compile a comprehensive list of partici- 
pants, then use a more general list, such as an overall 
extension newsletter list, mailing list, or Agricultural 
Stabilization and Conservation Service county register. If 
you use such general listings, you will have to make more 
than 40 contacts to locate 40 people who participated in 
the program you are evaluating. 

Modifying a Representative Sample 

If you are interested in learning how a program affected 
certain subgroups of participants, then you might con- 
sider purposeful sampling. For example, suppose a 
program with 100 participants had 10 dropouts. A repre- 
sentative sample of 40 participants woulfl, on the average, 
include 4 dropouts, too few dropouts to provide a reliable 
profile of all the dropouts. By interviewing all 10 
dropouts, however, you could make general statements 
about this audience subgroup. Any discussion of the 
overall findings of the study should be based on data 
from the 4 randomly selected dropouts only. 

Planning Aid E: Indicate your plans for selecting 
interviewees. (See page 6 of the PAP workbook.) 
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Step 6: Preparing an Interview 
Instrument 

One question that is often asked about a program's * 
results is the exteht to which they met program objec- 
ts, A program may or may not have objectives at each 
levels of reactions, KASA change, practice change, 
nd results. Even so, questions may be asked about 
suits of the program at these levels. For example, 
rdless of whether one of the objectives for your 
_ram was to increase participants' skills (level 5), you 
mfey need to answer questions regarding skill change. 

Although program objectives are only one basis for 
questibns regarding a program's results, you should try to 
include in your study those levels of evidence that corre- 
spond to ^significant degree with your program's objec- 
tives. 

Deciding Which Levels of Evidence 
to Study 

Before deciding which of the seven levels to include in 
your study, briefly consider the kinds of questions users 
might ask regarding these seven levels. 

Level 1— Inputs. What kinds of personnel and other 
resources, and how many, did extension expend on the 
program? 

Level 2— Activities. What kinds of information and 
methods of delivery did extension use to interact with 
program participants? 

Level 3— People Involvement. Who has participated in 
the program and how much? What have participants 
done in the learning situations provided by the program? 

Level 4— Reactions. How much have program activities 
appealed to participants? 



Figure 5 

Select the Levels of Evidence Most Needed 
by the Users of the RAP Study 




Level 5— Knowledge Change. How much have partici- 
pants changed their awareness, understanding, and 
ability to solve problems? 

Attitude Change. How much have participants' interests 
changed regarding the ideas or practices presented? 

Skill Change. How much have participants changed in 
terms of their verbal or physical abilities? 

Aspiration Changp. How much have participants 
selected future courses of action or made decisions 
regarding future courses of action? - 

Level 6— Practice Change. How much have participants 
applied their KASA change to their personal and working 
lives? (KASA stands for knowledge, attitude, skills, and 
aspirations.) 

Level 7— End Results. How much have participants and 
others been helped, hindered, dr. harmed by the results of 
changes in KASA and/or practices? 

Be careful that you don't select too many levels of 
evidence or the study may become too time consuming 
and complex- In general, select for the study only the 
levels that are insisted upon by a majority of the study 
users. (Figure 5 shows a hypothetical example of levels 
selected for a RAP study.) 

As the RAP team decides which levels of evidence to 
study, it should consider the following two factors. 

• How many levels can adequately be covered during a 
brief interview. 

• The amount .of time that has elapsed since the 
program. Participants set aspirations (level 5), change 
their practices (level 6), and experience end results (level 
7) over the weeks, months, and years following their 
involvement in a program. Thus, information on these 
levels of evidence can be obtained only after a significant 
portion of the participants have had the opportunity to 
apply KAS that they have acquired through the extension 
program. 



Planning Aid F: Indicate the levels of evidence 
that are most needed by the study users. (See 
page 6 of the workbook.) 
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Describing the Program 

For the first part of the interview instrument, you will 
have to prepare a half-page summary of the program's 
activities (level 2) and the people who were involved 
(level 3) over the time period being studied. This 
summary is to be read to or by the interviewee as the 
interview begins in order to: refresh his or her memory of 
the program; open up communication between the inter- 
viewer and the interviewee; and provide a common point 
of departure for the balance of the interview. 

Include "who, what, how, when, and where" as you 
prepare the description of the program's activities. 

• WHO conducted the program— extension, other agen- 
cies, volunteer leaders, etc? 



• WHAT Was the content or subject matter presented, 
discussed, etc.? 

9 HOW was the information communicated or delivered? 

• WHEN did the program take place? 

• WHERE did the meetings take place? Where was the' 
source for broadcasts? 

Include in your description of the people involved in 
the program: 

• Who participated— characteristics such as age, sex, 
occupation, and perhaps socio-economic status 

• How many participated— approximate numbers 

• How often—an estimate of the frequency of clientele 
participation and the amount of time they expended in 
participation 

• How intensely— as evidenced by participants' actions 
during learning situations. f 



Planning Aid G: Indicate your sources for 
evidence to describe RAP levels 2 and 3. (See 
page 7 of the RAP workbook.) 



Closed-End Items and Probe Questions 

The RAP interview instrument is composed of both 
closed-end items and probe questions. Closed-end items, 
unlike probe questions, provide the interviewee with 
optional responses from which he or she chooses an 
answer. The interviewer then checks the response cate- 
gory that most nearly corresponds with the respondent's 
answer. If the interviewee does not wish to choose one of 
the standard responses and wants instead to give his or 
her own response, the interviewer checks "other" in the 
interview instrument and writes in the answer the inter- 
viewee provides. 

We encourage you to follow up each closed-end item 
with a probte question, Probe questions help you define 
what an interviewee meant by a standard response — "to 
a fair extent/* "to a slight extent," etc. Probes also help 
check the validity of an interviewee's responses in that 
inconsistencies can become apparent between responses 
to closed-end items and probe questions. Such inconsis- 
tencies should be taken into account when the RAP team 
interprets the pattern of responses to closed-end ques- 
tions. 

Probes can be used to follow up all the closed-end 
questions, regardless of the level of evidence the question 
relates to. 

The following probe or nondirective questions can be 
used for any of the closed-end items: 

• Could you please explain. 

• Would you give me an example of what you mean. 

Validation and People Involvement 

As the intQ£yiew begins, the brief summary of the 
program's activities and the people who were involved in 
the program "sets the stage'* for the rest of the interview, 
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The interviewee reads, or the interviewer reads to him 
or her, the summary description of levels 2 and 3. The 
respondent then is asked to "validate" the description and 
then to describe the extent to which he or she partici- 
pated in the program's activities. You may want to use a 
standardized validation question or "item" to determine 
whether the respondent agrees with your description of 
the program. 

The following is a suggested validation item 

Say to the interviewee: 

Is this account of the (name of program) 

accurate as far as you know 

■ reasonably accurate 
not accurate 

don't know/don't recall 
other (specify) . — — 

The validation item confirms whether you have 
described the activities and the people involved in them 
accurately as the interviewee sees it, This confirmation 
ensures that you and the interviewee areJn fact talking 
about the same program and defining it similarly. 

You may want to use the following suggested item to 
determine the respondent's degree of participation in the 
program. 

Say to the interviewee: 

To what extent did you participate in/read or view 
(activity 1)? 

■ to a great extent 
to a fair extent 

: to a slight extent 

^ __ _ not at all 

____ don't know/don't recall 

■ other (specify) , — _ 

To what extent did you participate in/read or view 
(activity 2)7 
(Use the same response scale as above ) 

The items above identify any participants who believe 
they have not been in contact with the particular program 
you are studying and who therefore shouJd not be inter- 
viewed further. 

The following item can be used for parents of 4-H 
youth to determine their children's extent of participation. 

Say to the interviewee: 

7b what extent did your son/daughter participate 
in /read or view (activity t, 2, or 3)? 
(Use the same response scale as above.) 



Planning Aid H: Indicate what aspects of RAP 
levels 2 and 3 your interview will include. (See 
page 7 of the workbook.) 

The next step in constructing the interview instrument 
is to select or adapt RAP's standardized interview ques- 
tions regarding levels of evidence on program results, 
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Reaction Items 

* Use or adapt the interview questions below if you planr 
to obtain evidence on participants' reactions to the 
program. Refer to Planning Aid D as you "plug" an 
activity or activities into a prestated interview question, 

Say to the interviewee: 

To what extent did (activity 1, 2, and/or 3) meet your 
expectations at the time? 

— to a great extent 

. to a fair extent 

; to a slight extent 

^_ not at ali 

____ don't know/don't recall 

other (specify) - — ^ : — - 

If the interviewee answers "to a slight extent" or "not at 
all/' say: 
Could you please explain. 

Interviewee's explanation: - - — — - — _ 

There are two approaches that can be used to prepare 
reaction items: 

1 . You can prepare separate reaction items for each 
activity, one for activity 1, one for activity 2, etc. 

OR 

2, You can include two or more activities in one reac- 
tion item. 

f -. i 

You may wish to clarify the interview question 6t\ reac- 
tions as follows: \ 

To what extent did (method 1, 2, and/or 3) on (subject 
1, 2. and /or 3) meet your expectations at tfie time? 
(Use the same response scale as above.) 

An example of a reaction item for an interviewee who is 
an observer rather than a program participant might be as 
follows: 

Would you say that (activity 1, 2, and /or 3) met your 
son's! daughters expectations at the time? 
(Use the same response scale as above.) 

If you are eliciting reactions to several activities, you 
might consider listing the activities on the left of the inter- 
view instrument and the response categories across the 
page, and then checking the appropriate corresponding 
response. 

Planning Aid I: Indicate any aspects of REAC- 
TIONS that the interview will cover. (See page 8 
of the workbook.) 



KASA Change Items 

If you selected KASA change as a level of evidence on 
which to obtain information, use or adapt the items below 
for the interview instrument. Again, refer to Planning Aid 

a 

Knowledge Change Items 

Say to the interviewee: 

Think back to the activities in which you were involved. 
To what extent did you learn more about (subject 1,2, 
and/or 3)? 

to a great extent 

____ to a fair extent 
to a slight extent 

not at all 

' - don't know/don't recall 

J . other (specify) . — — 

("Other" here allows for any negative replies, such as: "I / 
received mostly misinformation.") 

If the respondent gives any of the first three responses 
or another applicable response, say: 
Could you give me an example. 

Respondent's answer: - 



You can vary the scope of the iterti(s) by using the 
options discussed in the section on reaction items: 

• Preparing separate items for each subject or content * 
area covered in the program, 

• Including two or three closely related content areas in 
one item. 

Remember, the subject or content areas covered in 
KASA change and practice change items can refer to a 
psychological or social process as well as a physical or 
biological subject. 

The above options for knowledge change items apply 
to all of the RAP items that follow, but for the sake of 
brevity they will not be repeated. 

The following is an example of a knowledge change 
item for an interviewee who is or was an observer rather 
than a participant. (This optional type of item will not be 
repeated, although it, too, is applicable to levels of 
evidence 5, 6, and 7.) * 

Say to the interviewee: 

Think back to all the activities in which your daughter/ 
son was involved. To what extent did he/she learn more 
about (subject 1, 2, and /or 3)? 

(Use the response scale and follow-up items above.) 
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You might also consider using the following item to 
detect the extent to which participants became more 
certain that what they already knew about a subject 
covered in the program was correct. 

Say to the interviewee. 

To what extent did your involvement make you certain 
that what you already knew about (subject 1, 2, and/or 3) 
was correct? 

(Use the previous response scale and follow-up items 
where applicable.) 

Attitude Change Item 

Say to the interviewee: 

To what extent did you become more interested in 
(subject 1, 2, and/or 3)7 

■ to a great extent 
to a fair extent 

to a slight extent 

not at all 

don't know/don't recall 

other (specify) _ — — — — 

Would you explain briefly what you mean. 

(An interviewee who responded M not at all" might 
explain: "I could not become any more interested 
because I already was extremely interested.") 

The expression "to what extent did you become more 
interested in," in the above item, could apply, for 
example, to a public policy education program to 
increase citizen awareness, interest in, and inclination to 
vote on a referendum. 

An alternate expression for detecting attitude change 
could be: 

To what extent did you become more favorable toward 
(subject 1, 2, and/or 3)? 

(Use the same response scale and follow-up item as 
above.) 

This alternative expression, "more favorable toward/' 
could be used for participants in an inducement, advi- 
sory, or advocacy-type extension program, such as a 
program designed to persuade families to use fluoridated 
toothpaste in an area in which there is no fluorine in the 
drinking water. 

Skill Change Items 

Say to the interviewee: 

To what extent did you acquire more skill in (subject 1, 
2, and/or 3)? 

. to a great extent 
_____ to a fair extent 

to a slight ejttent 

not at all 

don't know/don't recall 

other (specify) _ j — — — - 

(An "other" response could be: "I acquired a lot of skills 
that IVe never really needed.") 

if the interviewee selects one of the first three catego- 
ries, say: 

Could you give me an example or two. 

O 
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Aspiration Change Items 

Say to the interviewee: 

To whkt extent did you become more determined to try 
out (subject 1, 2, and /or 3)? 

to a yreaX extent 

to a fair extent- 

_____ to a slight extent 
not at all 

don't know/don't recall 

other (specify) — _ — — — 

(An "other" response could be: "I became less deter- 
mined because I saw that the idea does not apply to 

me.") 

If the interviewee selects one of the first three catego- 
ries, say: 

Would you mind giving me an example of what you 
mean. 

The expression "to what extent are you more deter- 
mined to try out," in the above item, pertains to kinds of 
actions that the interviewee has not engaged in previously 
(e.g., use of a home computer). 

An alternate expression for an item on aspiration 
change could be: 

To what extent did you becomp more determined to try 
out ideas on (subject 1, 2, and/or 3)? 

(Use the same response scale and follow-up item as 
above.) 

This alternate expression, "try out ideas on," is particu- 
larly appropriate for action that the participant has 
already engaged in but in a different way than recom- 
mended or offered by the program. 

Planning Aid J: Indicate any aspects of KASA 
CHANGE that the interview will cover. (See page , 
9 of the workbook.) 

Practice Change Items 

If you selected practice change as a level of evidence 
on which you will obtain information, and if the inter- 
viewee indicates any increase in KASA change, use or 
adapt the items below. (Refer to Planning Afd D for the 
subjects to plug in to the standardized items.) 

Say to the interviewee: 

To what extent have you put to use the ideas or skills 
you learned regarding (subject 1, 2, and/or 3)7 

to a great extent 

to a fair extent 

_ to a slight extent . 

not at all 

don't know/don't recall 

other (specify) _ — — 

("Other" responses to the question above or to a probe 
question could include: "I don't have the money to put 
these ideas Into practice"; "I haven't yet had the oppor- 
tunity to use these ideas or skills.") 
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Regardless of the response category chosen by the 
interviewee (except for "don't know/don't recall"), say: 
Would you please elaborate or explain. 

An alternative or supplementary item on practice 
change might ask: 

To what extent have you shared with others the ideas 
or skills regarding (subject 1, 2, anWor 3)? 

(Use the same response scale and follow-up items as 
above.) 

So far we have suggested using nondirective probe 
questions to follow the closed-end items. A directive 
probe could obtain even more specific information on the 
frequency and variety of use of content presented in the 
program. Here are two examples of directive probes. 

(a) Would you explain what you had in mind in giving 
your answer For example, how often during the past 
(time period) have you/your son/daughter usQd the skills 
or ideas regarding (conteny 1)? 

(b) Please provide an example or two— for instance, 
you/your son's/daughter s use of the ideas or skills 
regarding (content 2) at school, in community activities, 
or in fobs,. 



Planning Aid K: Indicate any aspects of PRAC- 
TICE CHANGE that the interview will cover. (See 
page 9 of the workbook. ) __ 



End Results Items 

Use the items below or modify them as needed if you 
plan to obtain evidence on end results, To use end result 
items, interviewees must have indicated some degree of 
KASA and/or (preferably) practice change. Refer to Plan- 
ning Aid D. 

Stay to the interviewee: 

you indicated that you have made use of the ideas or 
skills regarding (subject 1, 2, and/or 3). Overall, how 
helpful have the results been? 

____ ver helpful 
. fairly helpful 
. slightly helpful 
. no help at all 
. harmful 

. don't know/don't recall 
other (specify) 



(Interviewee's "other" response or response to the probe 
item might be: "It's too early to tqll what the results will 
be'T'l haven't made up my mind whether the results are, 
on balance, a help or a hindrance"; "I received some 
financial benefit"; "What we did has helped some 
members of the community but has had an unfavorable 
impact on others ") 



Regardless of the response category chosen by the 
interviewee (except for "don't know/donrt recall"), say: 
Would you please explain or give me an example or 

two. 

The closed-end item suggested above can be made 
more specific by plugging in various types of expected 
end results into an alternate expression of the item, such 
as: 4 

You indicated that you have made use of the ideas or 
skills regarding (subject 1, 2, and/or 3). How helpful have 
the results been in terms of (expected end result 1, 
expected end result 2, and for expected end result 3)? 

(Use the same response scale and follow-up item as 
above.) 

Again, use nondirective probes to help you obtain the 
interviewee's explanation for his/her selection of one of 
the optional responses. 

A directive probe can help obtain more specific kinds 
of explanations or examples regarding perceived end 
results An example of a directive probe on the financial 
results of a practice change is: 

About how much money have you gained, saved, or 
lost over the past (time period) as a result of using the 
ideas or skills from (Subject 1, 2, and /or 3)? 

Develop your own meaningful response categories 
(i.e., more than $2,000 lost; less than $2,000 lost; no 
losses, savings, or gains; less than $2,000 gained or 
saved; from $2,000 to $4,000 saved or gained, etc.). 



Planning Aid L: Indicate any aspects of END 
RESULTS that the interview will cover. (See 
page 9 of the workbook, ) t 
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Other Items and Procedures 

Although more time consuming and difficult to use 
than the type of reflective items above, other approaches 
can obtain more detailed evidence of perceived program 
results. You may wish to consider the approach of 
constructing lists of specific, possible program 
results and asking respondents to indicate which 
of these results apply to themselves. Also, you may wish 
to include several other items in the interview, such as the 
following open-ended question: 

What suggestions do you have for improving the 
program? 

.. . -'i ' * 

Estimating Input < 

If levef 1 was chosen fpr the purpose of studying the 
cost to extension of offering the program, first determine 
the total number of staff days expended on the program, 
within the time period covered by the study, for the staff 
from the county, area, district, and state levels. Then 
multiply the number of extension staff days expended by 
the average daily cost for each position. This will allow 
you to calculate the total estimated cost of the program. 
(Your state extension fiscal office should be able to 
supply you with the average daily cost for positions at the 
county, area, district, and state levels.) 

You may wish to include volunteer time expenditures, 
also. To do this, calculate the dollar cost of the program 
had volunteers been paid employees. This figure will 
represent the degree of savings gained through reliance 
on volunteer staff 



Biographical Data 

It is usually helpful to gather background data 
(biographs) during the interview. This assists in 
describing the interviewees as a whole and in analyzing 
the impact of the program on different types of partici- 
pants. 

The following information should generally be included 
in the biograph: 

• The participant's approximate age (i.e., 20-30, 30-40, 
etc.) 

• How much formal education the participant has had 

• The participant's sex 

You might also ask respondents for this information: 

• Occupation 

• Residence (city, suburb, town, rural community, farm) 

• Race 

• Family status 

• Other items as necessary 

Field-Testing the Interview Instrument 

The interview instrument should be field-testedwtoK, 
two or three program participants. Test and modify the! 
interview questions as necessary to ensure that \ 

• The evidence the users need is obtained. 

• Each item elicits accurate and complete information. 

• The interview is brief enough and interesting, as 
judged by the interviewees. 



Planning Aid M; Indicate any additional items or 
procedures that will be included in the RAP 
interview study, (See page 10 of the workbook.) 
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Step 7: Interviewing Program 
Participants 

Selecting Interviewers 

The next step of your RAP study involves planning who 
will do the interviewing and deciding how much training 
they will need. Interviewers may need to be trained to 
ensure that they each get equally accurate and complete 
information. At the very least, alMhe interviewers should* 
assemble to discuss the procedures for the interview and 
each of its questions. 

As you choose your team of interviewers, consider the 
following; ; 

• Will the interview data be more credible to users of the 
study if the interviewers are "non providers'- (no\ respon- 
sible for conducting the program)? 

• How many nonproviders are available to help conduct 
interviews? 

• How many ''providers" are available to conduct inter- 
views? 

• How much training will the interviewers (nonproviders 
and providers) need? 

One of the most important considerations in selecting 
an interviewer is whether the person has the confidence 
needed to contact people and pose questions to them. 
Having this confidence (Spends partly on whether the 
person feels it is appropriate^ ask program^ participants 
to provide interviews. 

Once you have selected a set of interviewers, assign 
them to the interviewees in a way that will minimize bias. 
You might consider doing this randomly. 

Planning Aid N: Indicate who will do the RAP 
interviewing, (See page 11 of the workbook.) 



Training the Interviewers 

The interviewers must be able \& establish rapport with 
the person they are interviewing, but at the same time 
must remain neutral The interviewer must not act more 
favorably toward the interviewee if he or she says favor- 
able things about the program's results. Likewise, the 
interviewer must not act shocked, angered, saddened, or 
embarrassed if unfavorable results are reported. (For 
further information on this, see Michael Quinn Patton's 
Qualitative Evaluation Methods. Beverly Hills. Catif Sage 
Publications, 1980,) 

Make sure that interviewers are willing and able to tak§ 
each respondent through the prescribed sequence of 
questions, asking each question as it is written. Inter- 
viewers should practice by interviewing each other. Prac- 
tice should be by telephone if the survey is to be done by 
telephone. 

The following three steps "set the stage ' for the inter- 
view. Have interviewers practice beginning the interview 
as follows: 

1, Interviewer introduces himself herself in an 
appealing way, 

2* Interviewer explains why the survey is being 
conducted. The interviewer could say: "Extension's 
success depends on meeting people s information and 
assistance needs. We are interviewing* people who have 

been in contact with the , program By * 

finding out how participants have been affected by this 
program, we expect to find ways for extension to do a 
better job/' 

3, Make the interviewee feel secure, The interviewer 
might say: "We are contacting all the people who partici- 
pated (or we^selected your nan\e through a chance 

drawing frorb a list of participants) in the 

program during the (time period). None of the informa- 
tion you give us will be released in a way that will identify 
you, Do you have any questions before we begin?" 



Planning Aid O: Indicate who will train inter- 
viewers and test the interviewing procedures. 
(See page 1 1 oMhe workbook ) 

Deciding on Face-to-Face or Telephone 
Interviewing 

Telephone interviewing is much more economical than 
face-to-face interviewing and usually is satisfactory for 
brief, standardized/open-ended interviews. Face-to-face 
interviewing provides more opportunity to establish 
rapport and communication through facial cues, etc., but 
telephone interviewing also has several advantages. 

1, It requires less of a time commitment from the inter- 
viewer and the interviewee. 

2 It is convenient for the interviewee, 

3. Sometimes the interviewee feels more free to express 
his or her opinions or report behavior. 

Face-to-face interviews may be necessary with low- 
income program participants, many of whom do not have 
telephones. 



ERIC 



Suggestions for Telephone Interviews 

1. Try to contact the interviewees by mail to prepare 
them for the telephone calls. Perhaps the chairperson of 
the county program planning committee could write a 
letter indicating the purpose of the interview, the approx- 
imate time of the call, and how long the interview will 
take. 

2. Consider sending a copy of the interview question- 
naire to the respondents to prepare them for the inter- 
view. If this is not advisable, expect to repeat some 
questions during the Interview, 

3. Because the voice is more important in teJephone 
communication than In face-to-face communication, 
voice modulation, diction, and appropriate pauses must 
be emphasized. 

4. Most of the interviews can.be done in the evening, 
but interviewers should not be ihtrusive and should offer 
to call back at a more convenient time if necessary. Two 
interviews per evening is a realistic target for most inter- 
viewers. 

Answers to the probe questions may be difficult to 
summarize unless you use a technique during the inter- 
view to classify answers. Consider precoding the subjects 
and the expected types of responses to probe questions. 
For example, precoded categories for probes regarding 
changing practices in family resource management could 
include: 

Subject 1— Long-range planning for family finances 
1a— Made plans for financing children's college educa- 
tion 

1 b— Made plans for retirement 

1c— Made plans for estate management 

As interviewees respond to probe questions, tally their 
responses, to the extent possible, according to the code 
you have developed (see Figure 6). 

It is not only the frequency of different types of 
responses to probes that is important. Some comments 
and suggestions express extraordinary insights or typify 
the results for various participants especially well. 
Recording these comments or suggestions verbatim can 
greatly enhance the quality and liveliness of the interview 
findings (see Figure 6). 

Planning Aid P: Indicate plans for interviewing 
procedures, (See page 12 of the workbook! _> 



Step 8: Analyzing, Drawing 
Conclusions and Evaluating, and 
Making Recommendations 

Analyzing 

To obtain a numerical profile of clientele-perceived 
results of the program, you should. count and record the 
number of interviewee responses in each response cate- 
gory of each item. 

Explore whether you can get the interview data 
computer analyzed through the state extension office or 
through a county extension office computer or computer 
terminal If you have fewer than 15 completed interviews, 
it may be simpler to analyze the data with tabulation 
' forms and a hand calculator. 

Tallies of the responses to the closed-end items will 
provide answers to questions such as: To what extent did 
the program's activities meet clienteles expectations? To 
what extent did clientele receive benefits or harm from 
KASA change or practice change? Consider converting 
tallies to percentages and bar charts. This will present the 
results of the program in a graphic, easy-to-understand 
way. 

You may wish to find out under what conditions the 
program had the most favorable results and the least 
favorable. This will help you and qfhers modify the 
program so tfcat its overall effectiveness will be enhanced. 
For example/ suppose interviewees who were involved in 
a program'sjfdemonstration activities indicated practice 
chanQearyo beneficial end results— two objectives of the 
program— but that those who were exposed only to other 
delivery methods of the program reported no such 
change or benefits. Such a finding might lead the RAP 
team to recommend emphasizing the demonstration 
method of delivery or recruitment of participants who 
would be receptive to demonstrations 

Interviewee-reported program results can be analyzed 
according to: 

• The characteristics of the participants— age, income, 
residence, size of farm operation, etc, 

• The delivery method or type(s) of activities that 
participants were exposed to 

• The subject matter that participants were introduced 
to. 

Consider presenting survey findings along the lines 
shown in the figure on page 17. Interviewees' "other" 
responses and responses to nondirectlve and directive 
probes should supplement and complement the analysis 
of the responses to the closed-end items. Use illustrative 
verbatim replies along with numerical profiles to present 
an overall picture of the frequency of similar kinds of 
responses 

Planning Aid Q: Indicate plans for analysis. (See 
page 12 of the workbook,) 
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Figure 6 



Participants' Ratings of Their 
Program-Induced Improvement 
of Skills In Weed Control 



Present RAP Findings as in This Hypothetical Example 



Size of 

participant's farm 



De gree of Improvement in Weed Control Skills 



500 acres or more 



(a) 
(b) 
(c) 



45% to a great extent 
35% to a fair extent 
20% to a slight extent 
0% to no extent 



(32) respondents 



55% to a great extent 
35% to a fair extent 
10% to a slight extent 
0% to no extent 

(35) respondents 

_1 ■ £ — ^ ^ • ■ ^ 

Examples of Illustrative and Verbatim Comments of 
Interviewees, Relative to Response Categories 

Response category (a), larger farms 
•Learned how to calibrate herbicide sprayer 
(5 respondents) 

Verbatim comment from 1 respondent: 'Through 
extension's conservation tillage program, I received 
v hands-on training in calibrating an herbicide sprayer/' 

•Learned how to identify noxious weeds (5 respondents) 
Verbatim comments from 2 respondents: "As a result of 
an extension workshop, I learned how to control 
multiflora rose in my pastures"; "I decided that 
extension's recommendations on chemical control of 
weeds are not as complicated as I thought they'd be " 

•No response to probe (4 respondents) 

V 

Response category (b), larger farms, etc, 



Less than 500 acres 



(e) 

(OCT 
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Drawing Conclusions and Evaluating 

The study's numerical tihdings--the percentages of 
respondents who answered the itemkln certain ways— 
should be converted to narrative findin§s~~N4rrative 
findings that could be based on the figure on page 17 are 
as follows: All the participants interviewed perceived that 
they had ga&ed at least some skill in weed control 
through their participation. About half the participants felt 
that they had made gains in weed control "to a great 
extent" rather than to a "fair" or "slight" extent, and those 
with smaller farms perceived greater gains than those 
with larger farms. 

To draw conclusions about the program's results 
requires interpreting study findings: findings have little 
meaning of their own. Insofar as the principal users of the 
study are going to eventually make decisions based on 
the findings, they should be encouraged to apply their 
particular perspective to drawing conclusions Irorp the 
study's findings. We therefore recommend that the entire 
,RAP team or a committee of the team inspect and 
IfmrpreX the findings. This group can examine the tables 
or charts, draw conclusions about the results of the 
program, and summarize these conclusions. 

In stating conclusions, it is probably wise not to attribute 
full credit for the practice changes and end results 
reported by interviewees to the KASA change that inter- 
viewees gained through program participation. In most 
programs, factors unrelated to the extension program 
have also influenced the participants. Most practice 
changes and end results of extension programs are 
•affected by such diverse influences as the weather, other 
sources of information, financial resources of the j 
program participants, and their motivations. It may be j 
judicious, therefore, to state! that ' the (name of the \ 
program) helped (proportion of the audience) to (change 
given praptfces with particular end results)." 

Conclusions about a program's results should be 
general statements about the results. Evaluations should 
be appraisals or judgments about how adequate or 
successful these results are. To judge a program's 
success or failure in terms of its results requires that the 
results be compared with established goals or standards^ 
Evaluation always demands an answer to the question 
"Success or failure as compared with what?" Thus, we 
suggest that the actual pattern of RAP findings be 
compared with an expected pattern. Furthermore, unless 
these expected findings are agreed upon before the 



survey is conducted, the team may be unable to agree on, 
an evaluation of the program's results. For this reason, we 
recommend that the RAP team (e.g., volunteer leaders, 
district agent, and others) join you, before the interviews 
are conducted, in specifying what findings would indicate 
that a program was "successful," successful, that is, in 
terms of interviewees* estimate of the results. 

The procedure of specifying the expected pattern of 
RAP findings will establish "exact objectives" against 
which participants' reflective evidence can be compared. '( 
The percentage of interviewees who are expected to 
select given responses can then be compared with the 
actual percentage selecting these responses, thus deter- 
mining the degree to which program results are judged to 
be adequate. For example, suppose a RAP team set the 
following criterion relative to the findings presented irl 
Figure 6: At least two-thirds of the interviewees should 
indicate that they gained skills in weed control "to a fair 
extlwit" or "to a great extent." Since over 80 percent of 
the interviewees stated that they acquired weed control 
skills "to a fair extent" or "to a great extent," the program 
could be judged to be quite successful as far as 
improvement in skills is concerned. 

Planning Aid R: Indicate procedures for drawing 
conclusions and evaluating the program's I 
results. (See page 12 of the workbook.) ; * 

1 

Making Recommendations 

To make recommendations regarding future programs, 
the study's conclusions and evaluation as well as informal 
evidence must bd taken into account. Thus, the evalua; 
tlon of a program's results based on reflective evidence 
has no automatic relationship to recommendations for 
future program finding or operation. For example, you 
may recommend (continuing an ineffective program long 
enough to find out whether certain modifications will 
work adequately. 

Before you recommend how a program can be 
improved, you should do the following, 

1. Determine the effectiveness of the program at 
different levels (e.g., practice change) by comparing the 
actual ,and the expected program results 

2 Select the level or levels that appear most in need of 
improvement, 

3, Suggest how to produce needed improvements. 

Program results at the higher levels are brought about m 
partly by results at the levels.below them. For example. 
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changes in practices are assisted by al significant degree 
of KASA change. Thus; if objectives for end results 
and/or practice change are not met arid those for KASA 
change are met, it may be necessary to either revise ihe 
objectives for KASA change or find out what barriers, are 
preventing participants from applying their acquired 
knowledge, attitudes, skills, and aspirations. This may 
mean deciding whether to initiate or recommend other 
types of programs, policies, and procedure^ related to the 
program. , J 

If KASA change is found to, be inadequate by program 
participants, the RAP team should consider^ 

• whether to modify the educational methodology and/or 
the content of the program or both 1! • , 

• whether to modify the intended audience or audiences 
for the program K 

• whether to modify the delivery system of the p^gram , 
internally or in relation to other organizatipns 

• whether to dtfer the way in which the program is 
managed (organizational structure and procedures) 

• whether to modify the program staff 

Consider alternative plans for improving the program 
and then recomnwnd which of these alternatives should * 
be adopted. If it isriot possible to make specific recom- 
mendations, state the issues, alternatives, or implications 
that should be considered. 

Recommendations of the type mentioned above relate 
to level 1 of ihe levels-of -evidence model ^changes 
recommendedNor levels 2 through 7 raise the question of 
whether to approve the same, more* or fewer resources 
for the program, The question of the amount of resources . 
to recommend for the program will exist, of course, 
regardless of the degree of success ascribed to the 
program by the RAP study. A successful program may 
thus be expanded if the need for the program still exists ^ 
or increases. Likewise, extension may decide that it 
should try harder (allocate more resources^ to improve a 
program judged not yet successful in terms of its results, 
On the other hand, a successful program may^eceive 
fewer resources because the program's very success 
makes its continuation less necessary, Also, the lack of 
success of a program may suggest that resources should 
be used more effectively in some other way, 



Sfep 9: Communicating the 
RAP Study 

. Study findings, conclusions, appraisals, and recom- 
mendations should be shared with decision makers in a 
way that will facilitate their decision making. 

Generally, there are two audiences for your findings, 
conclusions, and recommendations— stakeholders and 
the general audience. Stakeholders are those people who 
helped plan and conduct the study in order to obtain 
answers to their own questions, In the general audience 
are those people who became aware of the study and its 
findings only after it was under way or complete. 

With both of these groups your objective will be the 
same: .to encourage them to understand arvd use the 
study, The strategies for reaching these two groups and 
meeting your objective will, however, be different, Your 
final reporting strategies for stakeholders may be less 
important, since they will know about and begin to use 
the study's findings before the final report is completed. 
Nonetheless, you should explore with the stakeholders 
how the report should be packaged and presented so that 
it will be of the greatest help to them. / 

How to best share your findings, conclusions, and 
recommendations with general audiences or nonsponsors 
of the study (e g,, agricultural agency officers, bankers, 
, clergy, local business people) will require insight into 
their responsibilities, interests, and capacity to apply the 
study to decision making. 

Try to direct specif findings, interpretations, and 
recommendations to individuals who need stiph specific 
evidence. Bear in mind that different findings are likely to 
bejof interest to different individuals amTgroups. 



Planning Aid T: Indicate plans for^ommuni- 
eating the RAP study. (See Dag0^3 oi the work- 
book) ~f _ 



Planning Aid S; Indicate plans for helping deci- 
sion makers to' use the RAP study. (See page 13 
of the workbook \ ________ 



I» 



Additional copies of the RAP package may be ordered 
for $2 00 (New York State residents) or $2 50 (out-of- 
state residents) plus postage and handling. Minimum 
order $10 00, Address all inquiries to 

Media Services t 1 

B-10 Renssalaer Hall 
Cornell University 
Ithaca, NY 14853 
Tel : (607) 256-3157 
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Workbook 



for Planning a Study 
of the Clientele-Perceived 
Results of the 



Cooperative Extension program 
A 

S : — i : 

in County /Area 

Using Reflective Appraisal 
ol Programs (RAP) 

Completed J>y; 



(Extension Agent) 
and j 



(RAP Team Secretary) 



by Claude F. Bennett* 



0 



'Program Analyst. Program Development. Evaluation and Management 
Systems, Extension Service, U S Department of Agriculture. 
Washington, D C < A ^ 
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Pfenning Aid A: Choose a program for a RAP 
study. i 

Choose a program for a RAP study and jot down key words to describe the program 
according to the points below. These key words will help you write a more complete descrip- 
tion of the program later. 

Name or title of the program. 



Situation in the county, area, state, and/or nation that led to the development of the program, 
including needs, problems, opportunities, and capabilities. 



Program's objectives— immediate, intermediate, and long-range. 



Strategy, resources, activities, methods, and subject matter(s) of the program. 



ERIC 
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Extension personnel and volunteers responsible for the program. 



Other appropriate information about the program. 



Planning Aid B: Identify who might use the RAP 
study and for what purposes. 



Check (i/) the boxes that represent who might use the RAP study and for which purposes, 
(Check as many boxes as apply to the study.) 





Potential Purposes 


Potential Users \ 


To Imprpve 
Decisions 
on Programs 
or Program 
Resources 


To Improve 
Accountability 
Within Extension 


To 
Improve 
Extension's 
Accountability 
Externally 


To 
Improve 
Morale 


To 
Improve 
Understanding 
of 

Programs 


You/Other 4 
County or Area Agents 












Your Supervisor 
K ( District or Area) 












Your Program 
Development Committee 












Your County Legislators, 
County Court, or County 
Commissioners 






• 






Your 

State Legislator s 












Your State Specialist 
or Program Director 












Program Clientele 










\ 


Others: jl 










i 
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Planning Aid C: Indicate the individuals or 
groups you will invite to work with you. 

Check the individuals or groups you will invite to work with you in planning, conducting, 
and/or interpreting the RAP study. 

Council or program development committee(s) 

. Other extension agents 

Program or subject-matter specialist(s) 

Evaluation specialist 

j District or regional director 

* State/district program leader 
Other (specify) - — — — 



Planning Aid D: Indicate the scope of the RAP 

study, , _ 

List the activities, audience, methods, content, and time period that the RAP team will study. 
List also the types of reactions. KASA changes, practice changes, and end results that you will 
look for. (Fill in only those items that apply to the study.) 



Activities or types of activities you will study, 

1, _ - — — 

2. -j — — 



3, 



Audience(s) you will study. If you are studying more than one program stage, complete a 
separate workbook for the audience of each stage. 



2, 



Methods/you will study (the major delivery modes of the program) 

1. . — - — - — — — - — — 



2 



3. _ 
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Subjects or program content you will study (subjects can be biological, physical, economic, 
psychological, and/or social processes). 

1. , . 

2 _ - . : : . . _ 

3. ( . , _ 

4. __________________________ _ 

Time period of the program you will study— weeks, months, years (up to three). 



Reactions you expect to the program activities (how positive and/or negative). 



r 

Knowledge changes you will look for in terms of subjects (relative to the subjects you listed 
above). 

1. ■ . 

2. . ____ 

3. ; , , 

4. ________ 

Attitude changes you will look for in terms of subjects (relative to the subjects you listed above). 

1. „ . . . , 

2. ; , '. , .. 

3. , , : 

• 4 , : , , : , 



4$ 

o 
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Skill changes you will look for in terms of subjects (relative to the subjects you listed above). 

1. , — 

2. ; " . — • 

3. : - — ■ : : 

4. ' , . 1- , - - 

Aspiration changes you will look for in terms of subjects (relative to the subjects you listed 
above). 



1. 
2 
3 

4. 



Practice changes you will look for in terms of subjects (relative to the subjects you listed 
above). 



1. 
2. 
3 
4. 



f 



End results, i.e., consequences of KASA change and/or practice Change, you will look for, 

1. — . , . : . . . — ■ ■ — ■ ■ 

2 - - • -— ■ — ; ' 

3. , — — . — ■ : ■ 

4. , . , — •— • — — — 
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Planning Aid E: Indicate plans for selecting 

Interviewees. ■ 



Check the lists you plan to use to enumerate program participants and whether you will draw 
a sample of interviewee* Indicate the intended number of interviewees. 

^ Attendance lists 

„ Mailing list(s) 

Membership or enrollment lists 

List(s) of people who telephoned or wrote for advice, 

Other (specify) — — — — 

Based on the estimated total number of participants involved during the time period being 
studied, i.e., „ _ , it will probably be necessary to: 

interview all the participants 

interview a random sample of — participants 

interview a purposeful sample of — - participants 

The total number of interviewees is expected to be - — — . 



Planning AM F: Indicate the levels of evidence 
that are most needed by the study users. 

In the column at the left below, fill in the names or titles of people who vyill be users of the 
RAP study. Then check the levels of evidence that these people need. (Levels 2 and 3 are 
included in every RAP study.) Don't assume what evidence these people need Rather, find out 
which of the levels each user needs for understanding the program, decision making, 
accountability, and/or morale. Before you check the levels of evidence each person needs, 
find out whether this person already has sufficient or easy access to this evidence without a 
RAP study. If the answer is no, then you should check the box corresponding to that level of 
evidence. 



Which Ltvett of Evidence Are Needed by Whom? 




Ptertnlrtg Aid Q: Indicate your sources for 
evidence to describe RAP levels 2 and 3. 

What sources of evidence (to which you have easy access) can you depend upon to provide 
you with the information to describe accurately levels 2 and 3 (activities and people 
involvement) of the program? Check as many boxes as necessary. 



Levels of 
Evidence 


Source* of Evidence 


Extension Organization 


Self 


Personnel 


Records 


Memory 


Files 


(2) Activities 










(3) People 
Involvement 











Planning Aid H: This and the next few planning 
aids will help you determine the questions to 
include in the RAP interview instrument\ndicate 
what aspects of RAP levels 2 and 3 your \ 
interview will include. _ 

Check the Items below regarding levels 2 and 3 that you will include in the RAP interview 
instrument. 

A "validation item" 

Respondent s reflection on his or her degree of participation 

Parent s or other person's estimate of the respondent's degree of participation 

A nondirective probe (open-ended) item 



The items regarding degree of participation in the program will cover these activities or types 
of activities (should be the same as in Planning Aid D), 



Planning Aid I: Indicate any aspects of 
REACTIONS that the Interview will cover. 

Check those aspects of reactions, if any. that the interview will cover. s 

'. Respondent's reflection of reaction * 

r Parent's or other person's estimate of participant's reaction 

Nondireetive probe (open-ended) item 

The interview items on reactions will cover these activities or types of activities (will normally 
be the same as in Planning Aids D and H). 



1. 
2 
3. 
4. 
OR 



The items on reactions will cover these educational methods or delivery modes (same as in 
Planning Aid D) 

1. .. , . . .. . . _ . , , 

2. ..- . - - . : 



4, ■ „ _ . 

AND 

These subject matters (same as in Planning Aid D), 

1. , , , 

2. ! 

3. 



4. 




So 
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Planning Aid J: Indicate any aspects of KASA 
CHANGE that the interview will cover. \ 

Check those aspects of KASA change, if any, that the interview will cover. 

Knowledge Attitude Skill Aspiration 

Respondent's reflection of change — * — — — — - — — - — — — — — - — — — 

Parent's or other person's estimate — — — - — — * — — - — - — — — 

of participant's change 

Nondirective probe (open-ended) item - _ — : — — - — — — 

KASA change interview items, if any, will cover these subject-matter/content areas (same as in 
Planning Aid D). 

1. u- — • — — — ■ — ■ — — — — 

2. \ , — - - — ~— — — - — — 

3.. . — + — ~ — ■ — — — — 

4. . — ; — — — — — — — — — ^ 



Planning Aid K: Indicate any aspects of 
PRACTICE CHANGE that the interview will 

cover, , , 



Check those aspects of practice change, if any, that the interview will cover, 

- Respondent's reflection of change 
.' Parent's or other person's estimate of participant's change 

k - Nondirective probe « 

____ Directive probe(s) 

Practice change interview items, if any, will cover these subject-matter /content areas (same as 
in Planning Aid D) 



< 



Planning AM L: Indicate any aspects of END 
RESULTS that the interview will cover. 



Check those aspects of end results, if any, that the interview will cover, 



Respondent's reflection of end results that were a consequence of program-induced 
KASA change/practice Change 7 

Parent's or other person's estimate of end results that were a consequence of 
program-induced KASA change/practice change 
Nondirectfve probe 

Directive probe(s) * 



The interview items regarding end results will refer to consequences of these 
subject/content/practice change areas (same as Planning Aid D) 



1 

2 

3. 



The interview items on end results will cover the following expected consequences of KASA 
change and/or practice change (same as in Planning Aid D)> 



1 



2. _ 



Planning AM M: Indicate any additional items or 
procedures that will be included in the RAP 
study, 



Check any of the following additional items or procedures that will be included in the RAP 
stfjdy 

Interview items to elicit suggestions for improving the program 
Calculation or estimate of the costs of the program 
interview items to obtain biographical information on the interviewees 
A field test of the interview instrument 



Planning Aid N: indicate who will do the RAP 
interviewing. __ 



Indicate the approximate number of interviews that each of the following types of interviewers 
will conduct. 

_ County or area staff assigned to the program 

Subject-matter specialist^) '< , 

. Paid program assistants or paraprofessionals assigned to the program 

Volunteer extension leaders working on the program 

_ District director/program leader ^ 

County or area staff not-assigned to the program 

Paid interviewers 

■a Extension secretarial £taff » 



Program develojJfhent committee or council members. 
Other (specify) 



.« total number of interviews planned (should match number indicated in Planning Aid E) 



Fanning Aid O: Indicate who will train 
interviewers and test the interviewing 
procedures. 



Check who will train the interviewers, 



County agent 

District director/program leader 
. Subject-matter specialist 
Evaluation specialist 
Volunteer extensidfi leader 
Other (specify) , ■ .. 



Check who will be responsible for testing the interviewing procedures 

. County agent 
1 District director/program leader 
• Subject-matter specialist 

Volunteer extension leaders 

Qthftr (specify) . . — - - - _ - - ■ -« ■•- * -*^V 
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Planning AW P: Indicate plans for interviewing 
procedures, 

Check any of the following interviewing procedures that will apply to the Study s 

, . t ., . \. A coordinator for the interviewing effort will be named from the RAP team, 

A date will be set by which time all the interviews should be conducted, 

. j Guidelines will be set on the time of day (or evening) when telephone calls or personal visits will be made 
j ~ \3 Guidelines will be set on whether appointments will be made for conducting the interviews/ 

. , . m Guidelines will be set on how m*ny times an interviewer will attempt to reach an inteh/iewee 

Responses to the probe questions will be precoded 

. - Other (specify) ^ ^ . . , ,- ,. — - . » ■ ■ ■ - - - \ - — -■ ■ - ■ - — 



[ Planning Aid Q: Indicate plans for analysis 



Check the plans you will make for analyzing the interview data> ,, 

, A coordinator for the analysis effort will be named from the RAP team r* , 

Responses will be tallied according to analysis categories (check as many of the following Jhree categories as 
apply to this option): 

personal characteristics of the participants 
educational method paftteipants were exposed to 
content of the program participants were exposed to 
, Computer assistance for the analysis will be sought 
Response tallies will be converted to percentages. 
Tallies or percentages will be converted to bar graphs, y 

. Other (specify) „ — : — ! — . — — . — — - . - --■ — - •- ■■ - - - ••- 1 •• 



Planning Aid R: Indicate procedures for drawing 
conclusions and evaluating the program s 
results. 



Check the procedures that will be used to reach conclusions and evaluate the program s results 

; The RAP team will inspect the findings to reach conclusions or arrive at interpretations concerning what the 

results of the program were, 
„ ^ An appraisal or evaluation of the results of the program will be made 



The RAP team will evaluate the program s results by (if applicable): 

_____ employing whatevpf-cnteria each team member feels is appropriate after the findings fr6m th^ interviews are 
available. * * 
,„..„„ comparing actual findings of the study with expected responses recorded before the interviews are conducted 
Other (specify) ^,,-~., 7 , ,- - — , > - ~- - - - - - - - - - — - 
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j Aid S: Indicate plans for helping 
i makers to use the RAP study. 



Check your plans for making recommendations to decision makers or otherwise helping them 
use, the study. 

\ Issues regarding the program's future will be posed, 
. Alternatives for the program's future will be identified, 

Thft implications of selecting each alternative will be projected. 
One or more alternatives for the program will be recommended, 

A subgroup of the RAP team will pose issues, identify alternatives and implications, and make 
recommendations for review and acceptance by the entire team. 

Issues, alternatives, and/or recommendations will be directed toward decision makers external to extension, as 
well as toward decision makers within extension. 

Other (specify) . — — 



Planning Aid T: Indicate plans for 
communicafing the RAP study. 



Check the way(s) you intend to package and commodate the study findings, conclus^SW? r 
and recommendations . , 

Through special mailings summarizing or reporting on the study 
. Through a news release on the study 

Through regular reporting channels aimed at extension administration, county legislators, county 
commissioners, etc. 

Through meetings to brief selected audiences on the study 

By feeding-in findings, recommendations, etc. during program planning meetings, budget meetings, %tc 

Through a variety of brochures focusing on specific aspects of the study of interest to selected audiences 

Through volunteer extension leaders who will informally transmit the studyjs findings to county and state policy 

makers 

Other (specify) - '. - — — — 7 - — — — 
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